Welcome to TextPipe!

TextPipe: Online Help
 Welcome to TextPipe!

Submit feedback on this topic

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
What is TextPipe?
How can I use TextPipe?
Features
TextPipe in a Nutshell
User Assistance
How To Use TextPipe
Installation
Other documentation
Online Web Purchase

For new users

Remember - just press F1 for help at any time, or choose Index or Contents from the Help Menu.
  • TextPipe in a Nutshell
  • What is TextPipe?
  • What can I use TextPipe for?
  • How do I buy TextPipe?
  • Tutorial lessons, Quick Start guide, common tasks etc *Strongly recommended for beginners*
  • Frequently Asked Question1s and demonstrations (online)
  • Discussion forums (online)

For experienced users

  • An Introduction to Pattern Matching
  • Restricting filters to only operate in part of a document

For advanced users

  • Advanced Topics - automation and troubleshooting

For previous users

  • What's new in this version?

Updates and discussions

We're always adding enhancements to TextPipe. If you'd like to be notified when enhancements are available, please subscribe to the TextPipe update list below. We also have a discussion group for TextPipe users - an interactive forum for questions and answers.

We also run an extensive discussion forum.

TextPipe Update List
This newsletter that is notified of all updates and changes to TextPipe. Only Crystal Software can post to this list.

See also

Features
 More Reviews (online)

Contact Us Support Community Tutorials and User Guides (online)
© 1999-2005 Crystal Software . All rights reserved.


TextPipe in a Nutshell

 TextPipe in a Nutshell

Submit feedback on this topic

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home

To use TextPipe...

  1. On the Filters to Apply tab, tell TextPipe what to do by choosing filters from the Convert, Add, Remove, Email, Replace, Special, Map and Restrict menus
  2. Drag and drop your files or folders to anywhere in TextPipe's window. They will get added to the Files to Process tab
  3. Click the [Go] button.

Testing filters first...

You can test a filter by

  • Pasting text into the Trial Run Area, then clicking the [Trial Run] button

    OR
  • Clicking the File Output filter at the base of the filter tree, ensure that Test Mode is checked, then click the [Go] button.

Designing filters...

  • Click here for Common tasks and how to perform them

TextPipe's filter list is built up from a series of modular filters, each performing a small task. For example, the task of extracting email addresses from a group of files might look like this:

  1. Email menu\Extract email addresses - returns one email address per line
  2. Special menu\Merge - merge all the email addresses into one big file
  3. Special menu\Sort (case insensitive, removing duplicates)
  4. Remove menu\Remove matching lines (to remove your own email address, or known bad email addresses)

    To use these addresses in Outlook, you must convert them from one email address per line to a semi-colon separated list of email addresses.
  5. Convert Menu\End of line characters (convert from DOS format to the string ';')

To design a filter, you may have to reduce the problem down to a series of simpler steps. Use Special menu\Comment filters to document your design.

More resources

  • Tutorial lessons, Quick Start guide, common tasks etc *Strongly recommended for beginners*
  • An Introduction to Pattern Matching
  • Frequently Asked Questions and demonstrations (online)
  • Restricting filters to only operate in part of a document
  • Advanced Topics - automation and troubleshooting


What is TextPipe?

 What is TextPipe?

Submit feedback on this topic

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home

TextPipe™ is the fastest and most powerful text manipulation software you'll ever use. With TextPipe's easy to use interface you can achieve in seconds what would take you hours with other software. No two text processing tasks are alike - and only TextPipe can provide the flexibility and incredible text manipulation capabilities to get the job done.

TextPipe is a text processing application that takes a group of files and applies a set of operations or filters to each file in turn. Each filter performs an operation such as a search and replace, adding text to the left margin, converting end of line characters etc, and then passes the result on to the next filter, just as though the entire file had been processed using that filter first.

TextPipe has an extensive range of over 100 filters for adding, deleting, replacing, sorting and transforming text. Common tasks like converting files between PC, Macintosh, Mainframe and Unix formats are well catered for. With TextPipe's command-line automation or COM interface, complex processing tasks can be performed without user intervention.

Filters included can also split and join files, add line numbers, word wrap, convert between OEM and ANSI and remove duplicate lines/HTML/columns/binary characters/ANSI codes.

Web sites can be maintained easily with multi-file search and replace, and the ability to add standard text (such as copyright messages or banner advertising) to the start and end of each file. Multiple spaces and blank lines can be quickly removed to improve download times.

TextPipe replaces a substantial set of smaller text utilities with a unified and easy-to-use GUI. General users, web authors, administrators and programmers can readily perform complex text-processing with a minimum of technical knowledge.

Ziff Davis rates it as a 5 Star Editor’s Pick - "Despite all these bells and whistles, TextPipe is amazingly easy to use."

See also

Features
 New features in this release
 How can I use TextPipe?
More Reviews


TextPipe Tutorial

 TextPipe Tutorial

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up
Lesson 1 - Basics
Lesson 2 - Changing the Filter List
Lesson 3 - Moving and deleting Filters
Lesson 4 - Adding files
Lesson 5 - Restrictions and sub filters
Inserting data into a database

We recommend that you print out each lesson to make it easier to follow while using the program. To print this page, click the Print button above (and again as you get to each new lesson).

These Tutorials give you step by step instructions to introduce you to TextPipe.

  1. Lesson 1 - Basics
  2. Lesson 2 - Changing the Filter List
  3. Lesson 3 - Moving and deleting Filters
  4. Lesson 4 - Adding files
  5. Lesson 5 - Restrictions and sub filters

TextPipe for askSam

  1. TextPipe for askSam tutorial (online)

Advanced Topics

  1. Inserting data into a database
  2. Web site mining, mainframe reports and more (online)

Back to User Assistance Page


User Assistance

 User Assistance

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
TextPipe Tutorial
An Introduction to Pattern Matching
Regular Expression Hints
EasyPatterns
Sample filters
Step by step
Common tasks
Why isn't my filter working?

Remember, you can use the Filter Wizard [image] found under Tools Menu\Filter Wizard to easily create new filters.

What do you wish to do?

  • TextPipe Tutorial lessons *recommended for beginners
  • An Introduction to Pattern Matching
  • EasyPatterns
  • How to open the example filters
  • Step-by-step instructions for using TextPipe
  • See common tasks and how to tackle them
  • Advanced Topics - automation and troubleshooting


Features

 Features

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home

TextPipe can perform automated or manual line-feed conversion to and from DOS, Unix, Amiga and Macintosh input formats, and convert Mainframe fixed-length records (with a user-specified length) to separate lines.

To make files smaller and/or more readable, TextPipe offers filters to:

  • remove binary characters
  • remove ANSI escape sequences
  • remove blanks from the start and end of lines
  • remove blank lines
  • compress multiple whitespace to a single space
  • remove HTML and SGML codes
  • remove or resolve backspaces
  • change IBM drawing characters to + and | and -

To manipulate the text in files, TextPipe offers filters to:

  • perform an arbitrary mapping. The user can specify a string to be output for each of the 256 possible character codes. Strings can contain embedded control characters
  • search and replace: with case sensitivity option. It gives the length of search and replace strings -- especially useful for text replacement in binary files
  • add a prefix to every line (eg to add reply quotes to email)
  • change character case: UPPERCASE, lowercase, tOGGLE cASE, Title Case, Sentence case, rANdOm CAse
  • change character representation between ASCII and EBCDIC. Useful for mainframe interfacing
  • change between Windows OEM and ANSI. Useful with Windows and non-English languages

TextPipe offers the following file processing options:

  • can skip binary files. The user can also be prompted whether to process a binary file or not
  • can prompt before processing each file
  • can retain the original file's time and date
  • can recurse into subfolders
  • can process both text and binary files of any size
  • test mode where output goes to a new file rather than overwriting the original file

Special features include:

  • Filter lists can be saved and opened
  • Files can be quickly viewed with their associated application
  • Many methods of adding files to the list - selected via wildcards on the command line, specified in the application, drag and drop from explorer or via a Browse dialog
  • Accept a list filename on the command line to include large lists of files that couldn’t otherwise be specified. Can be used in conjunction with the Open List From File and Save List To File icons.
  • Accepts wildcards on the command line and in list files. The wildcards are expanded during processing.
  • No limit on input file size

TextPipe runs under Windows 95/98/ME/NT/2000/XP.

See also

What is TextPipe?
New features in this release
How can I use TextPipe?


Consulting Services

Professional Services

Crystal Software offers a range of Professional Services to our clients. Our services include:

  • Adding new features to our existing applications
  • Creating new applications
  • Creating customized, one-off solutions based on our existing applications
  • Consulting services to tailor the processing of TextPipe, ResumePipe, DataPipe and our other applications
  • Fixing and converting data files
  • Data mining web-based data
  • Fixing junk data in databases
  • Processing emails.

Typical projects that we have been engaged on include

  • Optimizing TextPipe filters for the greatest speed
  • Extracting data from websites to a database import file, or to an Access database
  • Reformatting corporate data to allow import or further manipulation
  • Removal of binary or garbage data from files
  • Writing VBScript/JScript code to perform custom file manipulations.

For more information, please click here.


What's New in v7.2
==================

*
Right-click menu for Replace field now provides easy insertion of captured
data as Tab or CSV-delimited fields or HTML.
*
The Prompt On Replace dialog now displays the length of each captured
expression.
*
EasyPatterns 1.9.1 - added [PipeField] and [Pipe] keywords.
*
Fixed error when manually choosing the Skip option to not replace text in a
DBCS file - more than one character was added. Only occurs under DBCS
versions of Windows (eg Chinese).
*
New error message when there are too many capturing expressions in a pattern.
*
Fixed bug with 'First row is header row' not being recognised for processing
delimited data with embedded carriage returns.

What's New in v7.1.7
====================

*
Added short cut menu items for converting between Unicode/ANSI and
Unicode/ASCII.
*
Added line number and offset properties to the TextPipe object available in
the scripting filter.
*
Map filters now have a comment property to describe what they are doing.
*
EasyPatterns 1.9 - added keywords for [number, numbers, leftAngle,
rightAngle, lessThan, greaterThan ]
*
Fixed restriction filters not supporting Secondary Output filters properly.
*
Fixed exact match filter not properly supporting Extract option, and not
correctly handling text when 'Skip' was chosen.

What's New in v7.1.6
====================

*
Search/replace lists now detect inserts, deletes and moves as changes. The
order of moved rows is now retained.
*
Fixed restriction filters not supporting Secondary Output filters properly.
*
Fixed EP keywords for [CRLF], [DOSNewLine], [EmailAddress] and [Hyperlink]
*
Fixed width font for Trial Run Input/Output is now correctly loaded.

What's New in v7.1.5
====================

*
EasyPatterns now allow repetition to be specified using the forms [2..3 ...]
and [2-3 ...] in addition to [2 to 3 ...].
*
Right-click menu item Escape Special Characters now works for EasyPattern
fields.
*
Fixed format string error on filter export.
*
Fixed Search/Replace list -caused an error when it was opened.
*
Fixed Remove Byte Order Mark (BOM) filter when handling UTF-8 files.

What's New in v7.1.4
====================

*
Added right-click menu to EasyPattern fields to make designing patterns
easier.
*
Added Line, Column and Length indicators to Search/replace filter.
*
A warning is now shown for trailing whitespace in the search text, which
often results in the text not being found.
*
Added 5 second countdown before TextPipe exits due to /Q command line
parameter.
*
Greatly improved performance for massive numbers of files- TextPipe no longer
checks for duplicate filenames.
*
Fixed EasyPattern for [month,monthnumber,day,hour] so that they match the
long version of the pattern first, then try the shorter variations.
*
Fixed range check errors with Compress packed/zoned decimal filters.

What's New in v7.1.3
====================

*
Fixed EasyPattern for [word] - previously matched non-word characters.
*
Fixed EasyPattern - '.' was not escaped - now fixed.
*
The Prompt On Replace dialog now shows the perl equivalent to an EasyPattern,
for verification purposes.

What's New in v7.1.2
====================

*
Fixed access violation with changes to the Restrict to Bytes filter.

What's New in v7.1.1
====================

*
Improved Restrict to Bytes (Byte x to End-y) to handle unlimited amounts of
data

What's New in v7.1
==================

*
You can now set the field delimiter for the Restrict To Fields, Copy Fields,
Move Fields and Delete Fields filters. This opens the way for semi-colon-
delimited, #-delimited, !-delimited, ^-delimited and other bizarre data
formats.
*
Added two new filters for compressing mainframe EBCDIC data to Packed Decimal
and Zoned Decimal formats.
*
The Split filter can now set the first filename number.
*
Added new Insert Bytes filter to complement the Insert Columns filter. This
new filter disregards line feeds when it adds text, and is very useful for
inserting commas into EBCDIC text as the text is converted.
*
New documentation links in the Filter Menus.
*
Synchronized scrolling is now properly restored when a filter is loaded.
*
EasyPattern fixes to align with the EasyPattern documentation
- [symbol, symbols] - removed the '!'
- [space, spaces] changed to just matched ASCII 32, not tabs, line feeds etc
- [horizontalwhitespace, hspace] - now only match space and tab
- [verticalwhitespace, vspace, worddelimiter, linedelimiter,
paragraphdelimiter] - now match formfeed
- [anybracket, anybrackets] - now match { and }
*
EasyPatterns now will NOT attempt to combine OR'd negated and non-negated
character sets, such as [ not lineChar or 'a' ]
*
HTML filter now matches tags when the tag name is immediately followed by a
tab or newline, not just a space.

What's New in v7.0.8
====================

*
Fixed EasyPattern handling of vertical tab (\v), which affects the keywords
verticaltab, verticalwhitespace, vspace, worddelimiter, wordchar,
columndelimiter, columnchar, linedelimiter, linechar, whitespace, word, line,
and column. This problem still exists in the perl pattern library, and can be
worked around by using \x0b instead of \v.

What's New in v7.0.7
====================

*
Expand Zoned Decimal and Expand Packed Decimal filters are no longer added
inside a Restriction filter.
*
Changed EasyPatterns for EmailAddress, HyperLink, PageNumber and DayOfYear so
that the constituent parts are captured into $variables for later
substitution.
*
Fixed EasyPattern for FormFeed.
*
Fixed handling of EBCDIC negative zoned decimals.

What's New in v7.0.6
====================

*
Fixed bug with parsing of text with \x{...} characters.

What's New in v7.0.5
====================

*
Fixed shell context items - the last item did not work. Also fixed the path
for the Split File filter.
*
Fixed bug with Rolling back .BAK files - if the Only Output Modified Files
option was on, it would delete original files.

What's New in v7.0.4
====================

*
Unicode conversion filter was only outputting the Byte Order Mark for the
first file.

What's New in v7.0.3
====================

*
Added swap button to Unicode conversion filter.
*
Fixed Unicode conversion filter bug.

What's New in v7.0.2
====================

*
Fixed Backup Mode behaviour when the .bak file already existed (now it is
deleted).

What's New in v7.0.1
====================

*
Fixed Hex Dump option for Edit Fields and File Grid.

What's New in v7.0
==================

*
New Unicode (UTF-16LE) search and replace, with options for space compression
and for ignoring non-spacing characters.
*
New Unicode (UTF-16LE) pattern matching.
*
Unicode conversion filter is now much simpler, as you just choose an input
and output format in one step.
*
New XSLT filter for manipulating XML.
*
New file grid right-click menu items:
- Analyze file - checks for various Unicode, XML, CSV and Tab delimited
formats, Unix/PC/Mac end of line format and other file types
- Hex dump - displays a hex dump of the first 200 characters of the file
*
Added \x{...} Unicode character matching for exact matches. This means that
for a Unicode space you can type \x00000020 instead of \x00\x00\x00\x20.
*
Synchronized scrolling option for Trial Input and Output.
*
You can now control if the Trial Input and/or Output are treated as Unicode.
*
Optimized single byte maps for the special case of one output character for
every input character.
*
Better help contexts for the 9 different search/replace types.

What's New in v6.9
==================

*
Removed restriction on pattern recursion depth - this was causing Stack
Overflow errors for legitimate expressions.
*
Fixed file grid ordering - if you moved rows around, TextPipe did not
remember the new row position.
*
Fixed menu items for Remove, Copy and Move tab fields. These did not add the
required filters.
*
Fixed help hyperlinks on Trial Run and Files tabs.
*
Fixed bugs in Unicode Make Little/Big Endian filters.
*
Fixed bug in Convert From Unicode filter, for when it encountered a character
that could not be converted. It now outputs a space.
*
The EasyPattern [year] keyword now checks for a 4 digit year before checking
for a 2 digit year.
*
Convert Word Document To Text filter now only closes the Active Document,
avoiding the MS Word error message 'This method or property is unavailable
because a document window is not active'.
*
Fixed line wrapping problem with the clipboard export text of several
restrict-type filters.
*
Increased maximum stack size to cope with increased size of Unicode filters
(and avoid Stack Overflows).
*
Fixed database filter not quoting char or date/time data fields.

What's New in v6.8.3
====================

*
New filters for handling pipe-delimited files:
   Remove pipe fields
   Restrict to pipe fields
   Copy pipe fields
   Move pipe fields
*
Added EasyPattern keywords for slash, colon, hash, pound, percent, star,
asterisk and ampersand. EasyPatterns now generate much better perl regexps.
*
EasyPatterns can now set the maximum expected match size - this allows them
to match text larger than the default of 4K.
*
Increased processing speed for Trial Runs and Clipboard output.
*
Restrict to fields filters now export the settings for Process Fields
Individually, Exclude Delimiter, Exclude Quotes.
*
Fixed EasyPatterns [month] and [monthname] to attempt to match a long month
name (January, February) first, then a short month name (Jan, Feb).
*
Fixed bug with Context Grep with lines after the match. Matching lines at the
end of the file were being omitted.
*
Fixed code for handling CSV, Tab and Pipe delimited files with header lines.

What's New in v6.8.2
====================

*
Added new EasyPatterns for [ CSVfield ] and [ TABfield ].
*
New Restrict to Byte Range filter - useful for expanding mainframe packed and
zoned decimal fields.
*
New Delete Byte Range filter.
*
XML filter now has better matching of double and single quoted attributes.
These filters now handle embedded double single or double quotes, quotes
escaped with backslash, and embedded new lines.
*
Optimized expand packed decimal and expand zoned decimal for speed.
*
Fixed bug with swap UTF-16 and swap UTF-32 word order.

What's New in v6.8.1
====================

*
Added Expand packed decimal and Expand zoned decimal filters.
*
Fixed handling of Restrict to between tags filter - under some conditions the
trailing > of the start tag would be put in the replace result.
*
Fixed Copy CSV/Tab fields bug. When copied fields were appended, the
delimiter was not inserted if the copy position was the next available field.

What's New in v6.8
==================

*
Added Unicode functions
   - Remove BOM (Byte Order Mark)
   - Swap UTF-16 word order
   - Swap UTF-32 word order
   - Make Little Endian
   - Make Big Endian
*
Added 144 Unicode conversion functions to convert to and from:

ARMSCII-8       ASCII          BIG5              BIG5HKSCS
C99             CES-BIG5       CES-GBK           CNS-11643-1
CNS-11643-15    CNS-11643-2    CNS-11643-3       CNS-11643-4
CNS-11643-5     CNS-11643-6    CNS-11643-7       CNS-11643-INV
CP1046          CP1124         CP1125            CP1129
CP1133          CP1161         CP1162            CP1163
CP1250          CP1251         CP1252            CP1253
CP1254          CP1255         CP1256            CP1257
CP1258          CP437          CP737             CP775
CP850           CP852          CP853             CP855
CP856           CP857          CP858             CP860
CP861           CP862          CP863             CP864
CP865           CP866          CP869             CP874
CP922           CP932          CP949             CP950
DEC-Hanyu       DEC-Kanji      EUC-CN            EUC-jisx0213
EUC-JP          EUC-KR         EUC-TW            GB18030
GB2312          GBK            GEORGIAN-ACADEMY  GEORGIAN-PS
HP-ROMAN8       HZ             ISO-2022-CN       ISO-2022-CN-EXT
ISO-2022-JP     ISO-2022-JP1   ISO-2022-JP2      ISO-2022-JP3
ISO-2022-KR     ISO646-CN      ISO-646-JP        ISO-8859-1
ISO-8859-10     ISO-8859-13    ISO-8859-14       ISO-8859-15
ISO-8859-16     ISO-8859-2     ISO-8859-3        ISO-8859-4
ISO-8859-5      ISO-8859-6     ISO-8859-7        ISO-8859-8
ISO-8859-9      ISO-IR-165     JAVA              JIS_X0201
JIS_X0208       JIS_X0212      JOHAB             Johab-Hangul
KOI8-R          KOI8-RU        KOI8-T            KOI8-U
KSC-5601        MacArabic      MacCentralEurope  MacCroatian
MacCyrillic     MacGreek       MacHebrew         MacIceland
MacRoman        MacRomania     MacThai           MacTurkish
MacUkraine      MULELAO-1      NEXTSTEP          RISCOS1
SHIFT-JISX0213  SJIS           TCVN              TDS565
TIS-620         UCS-2          UCS-2BE           UCS-2-Internal
UCS-2LE         UCS-2-Swapped  UCS-4             UCS-4BE
UCS-4-Internal  UCS-4LE        UCS-4-Swapped     UTF-16
UTF-16BE        UTF-16LE       UTF-32            UTF-32BE
UTF-32LE        UTF-7          UTF-8             VISCII

*
New context menu item to insert control characters like tabs and form feeds.
*
New context menu item to show a hex dump of the selected text.
*
Added logWarning to scripting model.
*
Added Cut/Copy/Paste to file grid context menu.
*
Added Tile function to Window menu.
*
The Convert Word Documents To Text filter now disables Word's screen display
for faster processing.
*
The TextPipe Engine DLL now has a COM interface identical to TextPipe's, but
removes most GUI operations for incredible speed.
*
Added alert() function to the internal TextPipe object of the scripting
filter, so JScript functions can easily display debug dialogs.
*
EasyPatterns bug fixes and improved flexibility. In particular, the [char]
keyword was fixed.
*
Several filters that should have allowed subfilters did not. Filters affected
were: database filter, insert column filter, add left margin filter.
*
Fixed bug - the filename was being output even for non-matching lines, when
context lines were included.
*
Fixed bug with partial trial run menu items including too many items.
*
Fixed bug with the filter tree allowing nodes to be pasted on the Input
Filter, resulting in error-producing subfilters of the Input filter.

What's New in v6.7
==================

*
EasyPatterns - English-like text patterns for mere mortals e.g. to match
phone numbers, use
   [3 digits, punctuation, 3 digits, punctuation, 4 digits]
As easy as that!
*
Many fantastic new HTML and XML filters!
* Restrict a search/replace or other filters to an XML element, attribute or
to text between XML tag pairs
* Remove an XML element, attribute or XML tag pair.
*
New Restrict To Filename filter. This allows you to search and replace inside
filenames, number and rename files and much more!
*
Very powerful new debugging feature - selective trial runs. You can select
which filters are being tested without having to clone a piece of the filter.
You can perform a Trial Run on:
* Entire Filter List (current behaviour, default)
* Selected Filter Only
* Selected Filter and Sub-Filters
* Filter List up to and including Selected Filter
To see these new options, right-click on a filter in the tree view.
*
New Backup Mode - renames original files to .BAK. Easily restore originals
with a single click in the Results window.
*
New Convert Word documents to text files filter.
*
Fixed bug when the Result window Cancel button was clicked twice in a row.
*
Fixed bug with pasting filter tree items that were copied from level 2 or
deeper.
*
Fixed bug with the Windows Explorer Extension that showed the TextPipe menu
multiple times.
*
Fixed bug with multiple consecutive Script filters not passing header text.
*
Fixed bug with Search/Replace Enable UTF-8 option and Extract option being
swapped when saved.
*
Fixed loading of Unicode maps from .TAB and .CSV formats - blank fields were
erroneously skipped.

What's New in v6.6.5
====================

*
Fixed bug with Remove Duplicates option of Sorting filter not being
respected.

What's New in v6.6.4
====================

*
New macros @time, @date and @datetime.
*
Fixed bug with handling of CSV fields with more than one line embedded in a
column.

What's New in v6.6.3
====================

*
Vastly improved sorting performance for files larger than 50 MB.
*
Simplified user interface by removing
1. Comments tab (old comments are now inserted into the filter list as the
first item)
2. Forcing the MDI child window to initially appear maximized, making it look
like a SDI application. The window size can still be changed as before
3. Removed the title that appeared above the properties
4. Removing the scratch pad tab unless a loaded filter has text in the
scratch area.
*
Added UTF-32 Unicode Maps
*
When Prompt On Replace is on, it can now be disabled during the search. Note
- this does not update the tree view immediately, and if the filter is
currently selected it has no effect.
*
The evaluation version provides a warning if the user attempts to modify
Excel or Word documents.
*
Unicode maps can now append multiple map files together into one map.
*
Unicode maps can now log overlapping regions as warnings or errors.
*
Improved Unicode map timings for populating with Default or blank values.
*
Added multi-maps for mapping input Unicode to multiple output Unicode forms
depending on command line parameters.
*
/MINIMIZED now examined before any other parameter, eliminating flicker.
*
Fix error when the file grid was selected in the askSam version.
*
Fix for Run Time version that prevented filter additions after a filter had
been loaded with /F=.

What's New in v6.6.2
====================

*
New File Grid context menu option to copy 100 Kbytes to the Trial Run Area.
*
Upgraded to PCRE 4.3 pattern matching library to fix assertions not working
correctly.
*
Added warning message that not all filter types can be exported to the
command line.
*
Tips and Wizard dialogs now allow filters being double clicked on from
Windows Explorer to be loaded.
*
Tips and Wizard are only shown if a filter hasn't been automatically loaded.
*
Run Time version now disables sound by default and does not show a tray icon.
*
Filter overlay images are now updated correctly when a disabled sub filter is
deleted.

What's New in v6.6.1
====================

*
Added Exit Codes depending on the highest level of error logged.
*
New /EXPORTFILTERTOLOG command line option.
*
Changes to support TextPipe Engine file processing.
*
Trial Input now preserves Unix EOLs when a file is pasted or loaded.
*
Changed menu structure to make it easier for new users. Removed some icons
from the toolbar.
*
Fixed Access Violations with Remove Lines Matching Perl Pattern filter, and
on some random occasions, with Perl pattern matches.
*
Fixed code generation of setPerl() statement.
*
Fixed an extra blank line being inserted at the start of the next file after
a Unicode file was processed.

What's New in v6.6
==================

*
New filter supporting search/replace lists. The lists can either be specified
in a grid, or they can be loaded dynamically from Excel, CSV or Tab-delimited
file formats at run-time.
*
The file list now supports multiple file specifications separated by semi-
colons eg c:\INetPub\*.html;*.txt;*.htm
*
The Split Files filter can now control how the output files are named.
*
Subtract entries in the file list are now shown in red, and Ignore entries
(comments) shown in grey.
*
The Script filter now uses a Syntax highlighting editor to display VBScript,
JScript, VBA, PerlScript and PythonScript code. It also allows separate
default scripts to be saved for each language.
*
Unicode maps can now be loaded from and saved to Excel/CSV/Tab formats.
Unicode filters are now far faster and can deal with undefined characters -
such as removing them, replacing them with a custom string, sending them to a
VBScript sub filter for specialized processing, or allowing them to pass
through unchanged.
*
The filter tree view now supports Cut, Copy and Paste clipboard operations.
There is also an undo action for the current filter. Filters can also be
pasted into any application as a text export.
*
CSV and Tab-delimited files with CR/LFs embedded in fields are now handled
correctly. New restriction filters can be used to remove the embedded CR/LFs.
*
CSV and Tab-delimited restricted filters can now remove the comma or tab
field delimiters and the surrounding quotes to make it easier for sub filters
to work with field data. Each field's data can now be processed individually
rather than as a concatenation of field data.
*
Byte map lines where the output is different to the input are now shown in
yellow. Byte maps can now scroll directly to the character you press on the
keyboard - which makes finding mappings easier.
*
In the audit log, merge file filters now summarize the changes made by
filters that follow them in the filter list. Split file filters also
summarize the changes made.
*
GUI and internal changes to support the new French translation.
*
New command line options /CLEARCOMMENTS, /CLEARFILTERS, /SAVEFILTER,
/LOGENABLE, /LOGMODE, /LOGTHRESHOLD, /PREPAREFILTERFORDISTRIBUTION,
/SAVEFILTER, /PLAYSOUND.
*
When loading a list of commands from a command file with /Z=<command file
name>, comments can now be included in the file using a leading ';'.
*
Speed improvements for the GUI and for filter execution.
*
Audit logging now shows status for more filters including Scripts and Maps.
The audit log text shown for search/replace filters now matches what is shown
on the screen for ease of reference.
*
Various COM scripting changes to support new features, and to allow DataPipe
to set the current directory.
*
Perl patterns now support UTF-8.
*
Convert end-of-lines filter now handles Mainframe CMS V- and F-format files
(downloaded to PC in EBCDIC format).
*
Removed Rich Edit Memo controls - they caused RTF codes in plain text to be
interpreted as fonts, and incorrectly added extra CR/LFs to Add Margins-type
filters.
*
The file list had a bug where it added file entries more than once if they
matched multiple file specifications in the same group.
*
Fixed bug with loading of single byte maps where the map was not actually
loaded, from both the menu and from the map sub-form. Fixed Reverse function.
*
Unicode Maps - fixed problem where changes to the grid were not detected and
saved.
*
Fixed bug with importing Excel and Tab-delimited search/replace lists.
*
Fixed bug with appending filters from the command line with multiple /F=
appearing in reverse order. Fixed filters such as /Hx not being added.

What's New in v6.5.1
====================

*
New filters - CSV to Tab-delimited, CSV to XML, Tab-delimited to CSV, Tab-
delimited to XML.
*
Fixed error message when egrep pattern matcher encountered a \000 (null).
Only perl patterns can handle nulls.

What's New in v6.5
==================

*
Search and replace lists can be loaded easily using the Load List from
Excel/Tab/CSV formats in the Replace menu.
*
The Task Bar Menu and Map Menu now display sub folders of the TaskMenu and
Maps folders, and sort all the filenames and folder names shown. This makes
it very easy to categorize these maps and filters into folders.
*
If a filter has one or more of its sub filters disabled, a red slash is shown
through its icon in the tree view. This makes it easy to find filters where
disabled sub filters are hidden from view. A disabled filter still has a red
cross through its icon in the tree view.
*
Custom patterns can now be defined in the file custom_patterns.txt in the
TextPipe.exe folder. The patterns can be added through the right-click menus
on every field.
*
Sorting can now handle files of unlimited size. Previously sorting was
restricted to using physical and virtual memory.
*
New sorting options for date and time, date, and time.
*
New right-click option for filter tree view to turn a filter into a sub
filter.
*
Matching/non-matching lines and other grep line based filters can now use
Perl patterns, egrep patterns, MS Word patterns and Brief patterns. The
addGrepFilter COM function has changed to support this.
*
Export to VBScript/JScript/web page now also exports the search replace
options such as greedy matching, allow comments, buffer size etc. The edit
distance threshold parameter has been removed from the addReplaceFilter COM
function, and new functions added for setPerl, setEditDistance and
setBufferSize.
*
Menus and tabs re-organisation, selection filters renamed to restriction
filters.
*
A fixed width font is now used for most memo fields.
*
Improved handling of malformed CSV files.
*
Fixed export of Logging:Append to Log. The wrong value was being exported.
*
Fixed generation of random files of specified size. All files were being
truncated at 8k.

What's New in v6.4.1
====================

*
Added "Content Type" to system registry to help web downloads of .fll files.
*
COM object now registers itself in the Running Object Table (ROT), and frees
itself at the correct times.
*
New COM properties Application.visible and Application.activeFilter
*
Fixed bug with moving nodes with sub filters.
*
The Task Bar menu now correctly runs filters for the second and subsequent
runs - previously the status window stopped it from working.

What's New in v6.4
==================

*
New filter to Remove a line range (in addition to the Head/Tail filter).
*
New filter to Remove CSV fields.
*
New filter to Remove Tab fields.
*
New filter to Move CSV fields.
*
New filter to Move Tab fields.
*
New filter to Move Columns.
*
New filter to Copy CSV fields.
*
New filter to Copy Tab fields.
*
New filter to Copy Columns.
*
File/Open command can now open multiple filters at a time.
*
Drastically improved the speed of 'head' filters and Remove All - they now
skip reading the remainder of the file no matter how large it is (a 'Head'
filter is used to extract the first N lines of a file).
*
New option to display help in an embedded window that is synchronized with
the current action.
*
Added filter export to VBScript and JScript - this can be used to very easily
create code for ASP or web pages or simple scripts.
*
Added COM interface to create Head/Tail filter, Run External Program filter,
Split filter, Show/Remove Duplicates filter, Secondary Output filter. Logging
functions, Input filter and Output filter settings can now be controlled with
precision.
*
COM functions are now error trapped to prevent dialog boxes appearing in
server versions. Error messages are captured to the ErrorText property.
*
The Script Filter's default script can now be set.
*
Fixed display issue where memo field containing multi-line text (such as the
search and replace fields) scrolled to the bottom rather than displaying text
from the top.
*
The command line text of the External Program filter now does not have double
quotes removed from it.
*
TextPipe now can paste more than 64K of text at a time into a memo control.
Previously controls could hold more than 64K of text, you just couldn't paste
that amount.
*
TextPipe now forces text paste operations to be plain text.
*
Speed improvements with string translation caching.
*
Wizard form now aligns correctly.

What's New in v6.3.6
====================

*
A new option to only output modified files was added to the Output filter.
*
A new Search option on the File Menu allows filters to be easily located.
*
The status window now shows how many bytes of the input file have been
processed.
*
A quick short cut is now to double click on any of the filename fields to
simply open the file (e.g. Single file output).
*
The Remove and Remove All buttons now work for all tabs (previously they were
not used for the Comments, Scratch Pad and Trial Run tabs).
*
When filters are being run from the command line, a new blank filter window
is no longer created - this means that the status of the already running
filter can be easily seen.
*
The Execute Program filter now deletes the Input and Output files when
processing has completed. This filter is commonly used for sorting very large
data files and other external processing.
*
The context grep filters Remove matching/non-matching lines and Retain
matching/non-matching lines have been re-designed so that they work exactly
the same as the Select matching/non-matching lines filters.
*
The /I command line option (recurse subfolders) now also applies to file
lists loaded using /L=<filename>.
*
Added new command line option /CLEARFILES to clear the file list. Also added
/S0-/S6 to handle different search/replace types.
*
Fixed bug with Remove/Retain Lines Matching Perl Pattern List, where all
backslashes needed to be doubled twice.
*
Fixed bug with none of the /C... command line parameters working.

What's New in v6.3.5
====================

*
Fixed problem with Extract Email Addresses filter removing one extra
character from the start of the email address when it started with mailto:
sendto:, email= etc.
*
Improved window behaviour when TextPipe is activated from COM - the main
window is now not visible unless it is specifically shown.
*
Improved COM handling - TextPipe exits correctly now that there are no
internal references to the Application object.

What's New in v6.3.4
====================

*
Fixed problem with saving Count Duplicate Lines and Remove Duplicate Lines
filters.

What's New in v6.3.3
====================

*
Fixed problem with Remove Patterns From List filter - was crashing.

What's New in v6.3.2
====================

*
Fixed problem with multiple output filters overwriting the same file.

What's New in v6.3.1
====================

*
Fixed sub filters not working for filters Add Text Side by Side, Add
Repeating Text Side by Side, Remove Lines Found In, Retain Lines Found In,
Remove Matching Lines Found In, Retain Matching Lines Found In.
*
Added new options to the Export Command Line of the Output filter.
*
Fixed problem with memos not displaying text properly.

What's New in v6.3
==================

*
Added new Find in HTML helper window to help users Edit, Remove, Replace or
Append to text inside HTML tags (such as the ALT tag of IMAGES), between HTML
tags (such as between <TITLE> and </TITLE>), and between two arbitrary pieces
of text.
*
Added new 'Select each line in turn' filter. This is a short cut for creating
a 'Select matching lines' filter with a pattern of '^'.
*
Window Menu now works properly - workaround found for Windows bug (being to
turn off AutoHotKeys and AutoLineReduction).
*
New System menu item (on task bar) to force TextPipe to close, even if it has
been activated via COM or from DataPipe etc.
*
Fixed double quotes appearing in filename fields after a filename has been
chosen.
*
When special characters are pasted, the default mode has been changed from
decimal to hex (so where \018 would previously have been pasted for special
character 18, now \x12 would be pasted)
*
New Word Wrap toggle item in the popup menu of memo fields.
*
New Start Column and Length options for Sort filter, Remove Duplicates filter
and Count Duplicates filter.
*
When a command line filter has been started with /G, TextPipe no longer
clears the filter list so that you can now see the contents of the command
line filter.
*
New command line parameter /OUTPUTKEEPSTRUCT to control whether the input
file's folder structure is retained when a new output folder is set.
*
Fixed a bug with dragging and dropping filters, where you could get an Access
Violation trying to edit the filter after moving it from inside its parent
filter.
*
Fixed @inputFilename and @fullInputFilename incorrectly using the output
filename.

What's New in v6.2.9
====================

*
Expanded the Special\Find and Replace menu item into an entire menu to make
it clearer.
*
Now uses Windows standard dialog boxes for easier translation.
*
Removed status window control resizing.
*
Removed validation on perl patterns that prevented back references from
working.
*
Fixed error with perl ^ not matching the start of the first line in files
less than 4K.
*
Fixed error with Extract Filter - some non-matching text was being passed to
the output.
*
Fixed the maximum allowed value on various filters - Split At Lines, Detail
lines allowed before writing to log file, size of random input file that can
be generated, insert column position, line numbering, context grep lines
before and after, delete columns values, duplicates filter, record length on
EOL filter, head/tail filter, all number-type filters like spaces to tabs,
centering etc, maximum buffer size for pattern matching etc.
*
Fixed handling of single and double quotes inside database queries.
*
Database filter 'insert script' option was not modifying file changed status.
*
Fixed TextPipe.getGlobalVar - now returns a blank string if the variable has
not been defined, instead of the variable name preceded by '@'.
GlobalVariables are now case-insensitive, and can hold multi-line values.
*
Fixed bug with 'Remove Prompting' not setting the modified flag if the filter
changed.
*
Changed the text qualifier for database insert scripts from " to '

What's New in v6.2.8
====================

*
All fonts are now set to the system default font. This makes translation of
Eastern languages far easier.
*
Tools\Put Sample In Trial Input now only includes files that have the 'Add'
action enabled. It also handles the situation where no files have been
specified.
*
Better DataPipe integration - now no longer hides the window when TextPipe is
closed. The main form minimizes instead.

What's New in v6.2.7
====================

*
Improved Command Line Wizard dialog.
*
New right click menu function for search/replace filters "Allow RTF tags"
converts the pattern to one that allows embedded RTF font tags.
*
Better integration with DataPipe - COM functions have changed, now
Application.hide minimizes TextPipe, and Application.show restores it.

What's New in v6.2.6
====================

*
Improved speed of COM functions.
*
Added COM code to support DataPipe
*
Fixed bug (again) where null lines caused the pattern matcher to fail.

What's New in v6.2.5
====================

*
Race condition fixed where a command line processing thread started with /G
/Q doesn't get a chance to start before the application exits.
*
New command line parameters /MAXIMIZED and /HIDDEN (hidden acts the same as
/MINIMIZED)
*
COM functions execute() and executeClipboard() now don't return until the
processing thread has completed.
*
New COM function isRunning to determine if a processing thread is still
running.
*
The COM function addReplaceFilter()'s ignoreCase parameter has been renamed
to matchCase - now the sense of the value is correct, and is consistent with
the user interface.
*
Fixed bug with perl-style patterns when used with look-behind assertions -
100 characters of text preceding the match is now retained in the search
buffer.

What's New in v6.2.4
====================

*
Secondary Output Filters can now be disabled and deleted.
*
Fixed errors occurring when a filter was modified after it had been moved up
or down using the up or down buttons.
*
Added Application.Quit function to COM
*
When created from COM, the default is now for the main form to be invisible.

What's New in v6.2.3
====================

*
Secondary Output Filters can now be added throughout the filter list to
duplicate the output file, or write selected lines and columns out to a
separate file, eg to place error lines or records in an error file, or to
separate records of different type or length into new files.
*
A new Select Block filter allows undelimited blocks of data to be selected
and processed as one.
*
Fixed VB/Jscript timeout - value is in milliseconds, not seconds.
*
Fixed obscure Perl replacement bug when run from command line.
*
Validation of ActiveX scripts now only occurs for VBScript - JScript
validation didn't work because the MS JScript engine does not implement
various functions.

What's New in v6.2.2
====================

*
Updated perl library to PCRE 3.7 (from 3.5)
*
EOL filter now allows more than one character in the replacement string.
*
Enabled some command line operations to provide Windows Explorer support in
TextPipe Web and Standard.
*
Fixed problem with the About dialog (in the evaluation version) interfering
with filter loading when TextPipe was invoked by double-clicking or opening a
.fll file.
*
Fixed bug with 'Escape Special Characters' not working properly.
*
Fixed bug with Split After XX Lines filter only outputting one line.
*
Fixed bug with 'Include Lines from End of File' when the number of lines was
greater than 255.

What's New in v6.2.1
====================

*
Tree view now displays more than just the first line of scripts and comments.
*
Fixed bug with perl-style grep and blank lines.
*
Input Filter export showed incorrect value for Delete Input Files After
Processing.
*
Filters loaded from the command line now have their full path saved, so they
get saved to the correct location even if the current directory is changed.
*
Added missing properties to Filter Export function.
*
Fixed bug handling command line filenames with embedded '..' paths.

What's New in v6.2
==================

Note: v6.2 was released quickly to correct some serious bugs, but it was not announced
until v6.2.1 was released with further bug fixes.

*
New filter to Remove lines matching any patterns in a list.
*
New filter to Retain lines matching any patterns in a list.
*
New filters to Retain matching lines, Remove matching lines, Retain non-
matching lines and Remove non-matching lines - this is easier to understand
than the Select lines filters.
*
New filter to add a file side-by-side with an existing file.
*
New filter to add a file side-by-side with an existing file and repeat the
added file if the file being added to has more lines.
*
New filter to Split a File after a given number of lines.
*
New Input Filter to generate random a data file of a particular size.
*
New filter to repeat a file's contents a specified number of times.
*
Added Script Timeout option to VBScript/JScript filter
*
Show Command Line dialog now also shows the current path and all environment
variables.
*
Added 'Escape special characters' menu item to right-click menus of all
fields. This is very handy for adding backslashes in front of pattern
matching meta characters, and doubling '@' and '$' characters where
appropriate.
*
Added 'Allow Extra Spaces' menu item to right-click menu of all pattern
fields, allowing you to easily search and replace in HTML files by ignoring
any extra spaces or tabs that commonly occur.
*
Added 'Allow Extra Line Feeds or Spaces' menu item to right-click menu of all
pattern fields, allowing you to easily search and replace in HTML files by
ignoring any extra spaces, tabs or line feeds that commonly occur.
*
Updated help, fixed application links to help file, help file now displays
when TextPipe is first run
*
Fixed bug with filter loading
*
Made splitter in Search/Replace filter visible, now resizes correctly.
*
Fixed 'Add folder' button - newer versions of Windows caused this dialog to
appear behind the main window.
*
Unknown field codes (a name preceded by an '@' symbol) are now NOT replaced
with a blank - they are left as is to prevent inadvertent use of a field code
causing text to disappear. Known field codes (@clipboard, @inputFilename,
@fullInputFilename) still work as normal.
*
Added new COM function to filter windows - closeWithoutSave. Added new
options to COM interface, including default values.
*
Fixed bug with a filter validation function not being called when there were
sub filters anywhere above them in the filter list.
*
Fixed a bug affecting the Select Matching Lines, Select Non-matching Lines,
Select Lines From Length To END, Select Lines From Start To End, Select Lines
From Start For Length filters when immediately followed by a line-based
filter such as a Reverse Each Line, Pad To Width etc. Some text was not
output at all.
*
Fixed a bug affecting the Remove Lines Found In List and Only Keep Lines
Found In List filters - because the list was not internally sorted, the
actual lines would only occasionally be processed correctly. A workaround was
to ensure that the line list is first sorted in ascending alphabetical order.

What's New in v6.1.1
====================

*
Improved the email address extraction filter to automatically remove leading
'mailto:', 'mime:', 'smtp:', 'email=', and remove trailing '?Subject' and
trailing periods.
*
Fixed Perl Pattern Match not saving 'greedy' setting
*
Fixed hidden form problem with active field being invisible
*
Fixed Access Violations on editing filter list

What's New in v6.1
==================

*
PERL-STYLE pattern matching! - use both greedy and non-greedy matching
options, as well as embedded comments, assertions, non-capturing sub-
expressions and more!
*
Brief-style pattern matching
*
MS Word style pattern matching - use the *, ?, etc wildcards that you can use
in MS Word.
*
New 'Extract' option for search and replaces, all non-matching text is
discarded.
*
The Found Text can now be referred to in the Replacement Text as '$0' for
Exact, Sounds-like and Edit-distance matches. Previously the Found Text could
not be referred to, making some replacements harder.
*
You can now easily add a sub filter to the selected filter by holding [Alt]
down while choosing it from the menu or selecting its speed button.
*
You can now easily add a filter at the parent level of a sub filter by
holding [Shift] down while choosing it from the menu or selecting its speed
button.
*
New Input Filter setting for text to come from the Trial Run Input area.
*
The filters Add Left Margin, Add Right Margin, Add Header and Add Footer now
do not parse the included file for special functions or characters. This
ensures that the file being included is included exactly as is. The parsing
of text specified in the edit field continues as previously.
*
Fixed a recursion bug when [Alt+Down] is pressed to move a filter and its sub
filters down, now the filters are moved down correctly.
*
Fixed bug with base panel of search and replace filter disappearing after
returning from the Trial Run area.
*
Removed the Font Control context menu item for controls.
*
Better handling of Cancel button during a processing run.
*
Fixed handling of NULL values with database filter and Generate Insert Script
mode.
*
Database Command Timeout setting for queries that run for more than 30
seconds.
*
Improved Tree handling is quicker and easier to use; the treeview changes
immediately when a setting is changed, and there is a new Up and Down button
to easily move filters up and down (in addition to Alt+Up and Alt+down).
*
All command line defaults now come from the Normal Template normal.fll, not
the internal defaults which used to be:
  inputFromFiles, promptOnEach=false, skip binary files,
binarySampleSize=100, outputToFiles, testMode=false, retainDateTime=false,
changeExtension=false, changeBaseDir=false.
*
Fix for the error " The Decision Cube Capacity is Low. Please deactivate
dimensions or change the data set"

What's New in v5.6.3
====================

*
Uninstall now removes all registry entries
*
Installer now supports unattended installs
*
Fixed short cut conflict between Remove/Remove All buttons and menu items
*
Fixed bug with registry security in registered version

What's New in v5.6.2
====================

*
Added 'Normal Template' functionality - a filter that is always loaded to set
defaults.
*
Delete Input Files after Processing setting now displays a warning box when
it is enabled. When a new filter is created this setting always defaults to
disabled.
*
Added shortcuts to tabs on status form, query replace form and command line
display form.
*
Fixed error in COM documentation for AddFile function - constants should be
1,2 and 3, not 0, 1 and 2.
*
Changed sizes of GUI controls to show foreign languages better.
*
Fixed bug with loading of foreign language, and enabled support for radio
group type controls.
*
Fixed bug with Append Mode.

What's New in v5.6.1
====================

*
Clicking on the filename column in the file list now sorts the filenames.
*
Improved COM interface - new methods for compiling the filter list which
improves speed if the same filter is used repeatedly
*
New Append to existing file setting

What's New in v5.6
==================

*
Localisation complete - now everything in TextPipe can be translated to a
foreign language by translating the file Languages\English.ini

For more details on how to translate TextPipe, please visit:
  http://www.crystalsoftware.com.au/textpipe/translate.html
*
COM interface now allows TextPipe to be fully controlled from languages such
as VisualBasic, VBScript, C++, Delphi etc

For more details on how to translate TextPipe, please visit:
  http://www.crystalsoftware.com.au/textpipe/com_interface.html
*
Fixed bug with dragging and dropping filters onto a filter that doesn't
support subfilters - now adds the dragged filter above it

What's New in v5.5.2
====================

*
Added code for TextPipe Pro Run Time Engine
*
Marketing changes for evaluation edition
*
Began preparation for localisation
*
Fixed command line processing bug with modifying the Output filter settings
after loading a filter with the /F= command
*
Fixed bug with debug windows

What's New in v5.5.1
====================

*
Forms and menus can now be translated into other languages via a .ini file in
the Languages folder
*
Improved the load time of massive filter lists loaded via the /Z= command
line parameter
*
Trial Output now shows Column and Row in the same way that Trial Input does
*
File size for files larger than 2GB is now displayed correctly
*
Improved the way that form sizes and positions are saved
*
Improved the way that registered user data is saved under Win2K
*
Forced installation to require the Administrator account to prevent registry
writing problems under Win2K

What's New in v5.5
==================

*
Lines of context surrounding a matching line can now be displayed, both
before and after matching or non matching lines (grep filter)
*
Added short cuts to tabs (Alt+1 through Alt+5)
*
Added Filter Wizard to help new users
*
Added command line options for setting the output file extension, output
folder, and merge output filename.
*
Sub filters can now be created by holding down Shift as they are drag and
dropped to a location inside another filter
*
The database filter now directly supports creating a database insert script
from extracted data
*
Improved tutorial lessons
*
TextPipe tutorial now listed in the start menu
*
Fixed bug with default folder not being TextPipe's folder (default was My
Documents folder)
*
Fixed bug with access violation on exit after a filter is invoked from the
command line
*
Fixed bug with Count Duplicates filter missing some lines
*
Fixed bug with Extract Emails filter missing the last email address if it
ended at the end of the file
*
Fixed bug with errors not being reported during trial run - now if errors are
reported the status dialog will not automatically close
*
Changed default extension to .fll for link filter dialog. Also added an open
button so that the filter can be easily loaded and edited

What's New in v5.4.1
====================

*
Can now CHANGE THE REPLACEMENT TEXT during a run. This is very useful for
fixing up mistakes in the replacement text, and can also be used for pattern
matches
*
New Paste Special command on context menus to allow pasting of special
characters without TextPipe converting them to its internal code
*
Unknown '\' entries are now output with the leading '\' rather than without.
This is to help new users
*
Trial Input and Output no longer attempt to validate special characters such
as \t etc.
*
Fixed bug with '\' being incorrectly interpreted in the Prompt for replace
window
*
Fixed bug with filters becoming modified when Search/Replace filter was
selected (not edited)

What's New in v5.4
==================

*
VBScript/JScript code can now control TextPipe - LogError, LogInfo,
inputFilename, fullInputFilename, terminate, setGlobalVar, getGlobalVar,
subFilterEntireText, version
*
VBScript/JScript code can now have subfilters. Subfilters can be turned on or
off by the script at any time using InitSubFilters, FlushSubFilters,
subFilterChunk
*
Find/Find Again in all memo fields (multi-line edit)
*
Font control dialog on all edit fields (including memos)
*
Extended the number of filters than can be invoked via the command line
*
Facility to generate TextPipe command line parameters from an existing filter
list
*
Trial Run can now be interrupted - runs in separate thread
*
Ctrl+A (Select All) now works in all fields
*
More options in the Window menu
*
Adhoc Transform tab renamed to Trial Run tab
*
Task bar menu now sends TextPipe to the back after a menu item is selected
*
Better handling of maximized windows when TextPipe restarts
*
Fixed bug with Select lines filter - if the input had (say) 2 lines and the
setting was from line 3 to END

What's New in v5.3.9 Beta
=========================

*
Added drag and drop between filter windows
*
Added drag and drop promotion/demotion and movement of filters in the filter
tree display (and removed the Up/Down buttons)
*
Added Lesson 4 to tutorial

What's New in v5.3.8
====================

*
"x2" function now duplicates sub filters at the same time
*
Fixed bug with Insert Column At filter where the column was set to zero (ie
insert at end of line) and the actual line length = 1 (the text was inserted
at the beginning of the line)
*
Fixed bug with replace filter and whole words option - output text can be
duplicated
*
Fixed bug with merge filter and single file output filter
*
Fixed bug with Close All function not closing all windows
*
Fixed bug with Insert Filter File function changing the name of the currently
loaded filter

What's New in v5.3.7
====================

*
Updated Export To Clipboard and Print facility to cope with new filters and
tree structured filters. This facility is an excellent way to maintain
version information about your filters
*
Maps now moved to separate 'maps' folder
*
textpipe.log is now created in the same folder as textpipe.exe if no path has
been specified
*
Recent Files menu now uses submenu of main menu
*
Filters opened will now load into the New Filter window if it is unchanged
and does not hold a filter
*
Fixed problem with files and folders added via the command line not
defaulting the subfolders setting correctly
*
Fixed problem with filter opening twice when opened from Explorer when
TextPipe isn't currently opened
*
Fixed bug with merge filter not performing any filters following it in the
filter list

What's New in v5.3.6
====================

*
Added ability for Windows Explorer to detect TextPipe and open a filter in an
existing TextPipe window
*
Recycle bin functionality temporarily removed while bugs fixed

What's New in v5.3.5
====================

*
Fixed buttons and help for Files Tab
*
Delete file operations now use the Recycle Bin
*
Fixed bug with Remove Lines From File and Only Show Lines From file filters

What's New in v5.3.4
====================

*
(Hopefully) fixed long-standing error on exit

What's New in v5.3.3
====================

*
New subscribe facility on help menu
*
Fixed regexp bug with characters sets [] involving ranges of characters
specified with a \ eg [\000-\255]. Also error message for character sets
containing a \000.

What's New in v5.3.2
====================

*
Better error handling when file delete/rename fails
*
Fixed bug with number of subfolders to process (1 Level was same as None)
*
File\Open now always opens a filter in a new window. New menu item to open in
the current window
*
Fixed bug with Remove button, only removed first file from file list even if
multiple files were selected
*
Now checks if log file exists before using Notepad to open it
*
Double quoting of filenames and folders containing spaces now handled
correctly
*
Better handling of dropped files

What's New in v5.3.1
====================

*
Added option to delete input files once they are processed - to support
integration with GFI FAXmaker

What's New in v5.3
==================

*
New Sorting options
* ASCII Sort (case-sensitive and case-insensitive). Previously only an ANSI
sort was available (where lower case letters sort before upper case letters)
* Numeric sort
* Sort by length of line
*
New Unicode Map filter for translating Unicode files
*
Improved online help with Quick Start guide, improved tutorial and more
cross-referencing
*
Status form now has indicator to show whether test mode or overwrite mode is
active
*
All memo fields now can save to and load from a file
*
Trial output now has warning message if the output text contains null
characters
*
Improved Command Line Assistant dialog
*
Fixed bug with Query Replace window not scrolling found text into view
*
Fixed bug with "First Match Only" option discarding text
*
Fixed move filter up/down bug with filter lists containing more than 256
filters
*
Fixed bug with database filter not extracting data correctly when fixed width
format with no column titles was performed
*
Database filter now correctly handles non-SELECT queries that return a record
set
*
Fixed bug with access violation on exit
*
Fixed bug with pattern match ignoring First Match Only setting under some
circumstances

What's New in v5.2
==================

*
Added help toolbar for easier access to helpful features
*
Improved drag & drop behaviour for .fll files
*
Added automation check to disable debug filters
*
Added better error handling for MIME decode quoted printable filter
*
Advanced logging details in results window
*
Added error checking to ensure file can be deleted/moved to correct final
filename, or error logged if not
*
Added output option for all files merged to a single file
*
New Clear All and Paste context menu item
*
Prompting dialog for Read Only files, Prompt Before Processing and Prompt on
Binary Files now has option for No To All and Yes To All
*
New Search/Replace option to only replace the first occurrence
*
Trial Input/Output now have horizontal scroll bars
*
Grep filters (Keep matching lines, remove matching lines) now as select
filters, where matching lines get subfilters applied to them
*
ActiveX scripting no longer requires dummy parameters for functions
*
Improved match information tab for Edit Distance and Sounds-like matches
*
Context menu on tree view for faster deletion/enable toggle
*
Database query now creates correct XML document. Handling of previously
unrecognised data types. Original query output as comment. Headers always
output even if 0 rows returned. Unique column names generated for all output
types. Encoding="UTF-8" removed from XML header - not supported by some
versions of IE
*
Status dialog now shows count of Non-binary Files Skipped
*
New Edit Distance option for search/replace filters - very handy for finding
and locating misspelt words
*
New option to retain the existing file list when opening a new filter
*
New facility for search/replace to replace all or none within the current
selection
*
Loading a .fll filter file now opens in the same window
*
Better detection of filter list changed
*
Better handling of loading very old filter lists
*
Added force lines to width filter
*
Added advanced logging facilities with detailed tracking of all changes made
*
Fixed exception with exact match filter
*
Fixed handling of non-select statements in Database filter, and improved
error checking code
*
Fixed directory where preliminary output file created
*
Fixed split filter - was not performing "start file" processing for following
filters
*
Fixed Remove Email Headers - was crashing
*
Fixed bug that affects all line-processing filters - when using Mac files the
first character on each line may be repeated
*
Fixed bug with exact match filter that caused text to be output multiple
times
*
Fixed Win NT bug with popup context menus on memo fields
*
Fixed delete columns filter
*
Fixed handling of grep/inverse grep filters
*
Comment filters now have vertical scroll bars for comment text
*
Fixed subfilter bugs where filter list was not built correctly, missing
several filters
*
Fixed bug with Select Lines filter not outputting its results
*
Fixed bug with Select CSV Fields filter
*
Fixed errors with start and end word for Edit Distance and Sounds-like
filters
*
Fixed split filter
*
Fixed handling of Select Columns with the 4th option
*
Fixed debug filter error
*
Fixed Select CSV/Tab field missing last field
*
Fixed handling of map menu and task bar menu

What's New in v5.1
==================

*
** Sub filters - drag and drop a filter onto another in the tree view and the
dropped filter will then process JUST the output from the parent node. Eg
drop a "Title Case" filter onto a "Search/Replace" filter to change the
search result to Title Case
*
** New Database Connection filter. Attach to any ADO/ODBC database and
execute any SQL command text against it. The result can be output as CSV,
Tab-delimited or XML (with automatically generated DTD schema). All
characters are appropriately escaped, including quotes within strings for
CSV, and XML special characters.
*
Pattern matching now extended to handle up to 36 ($0-9,a-z) bracketed
expressions. This is 26 more than any other product on the market.
*
New Select CSV Fields filter - select a group of consecutive CSV fields and
apply sub filters to the result by dragging and dropping them onto this
filter. Handles commas embedded in quotes etc
*
New Select Tab-delimited Fields filter - select a group of consecutive tab-
delimited fields and apply sub filters to the result by dragging and dropping
them onto this filter
*
New Select Line Range filter - select a group of consecutive lines and apply
sub filters to the result by dragging and dropping them onto this filter
*
New Select Column Range filter - select a group of consecutive columns and
apply sub filters to the result by dragging and dropping them onto this
filter
*
New Remove All filter - useful for subfilters
*
Convert EOL filter can now specify any string as the new EOL character
*
New Truncate to Width filter
*
File output has new "Maintain folder structure" option when outputting to a
new folder
*
Remove Columns filter now has more advanced options
*
New option Skip Prompt if Identical for Search/replace filters
*
ActiveX scripts can now easily output text at the start and end of each file
*
New tools added - option to verify a filter before it is run - performs
syntax check etc. Also added Windows calculator
*
Explorer extension - right click on files, folders or drives brings up
TextPipe options. New window on Tools menu to customise the Explorer
extensions
*
Command Line Assistant window - provides examples of how to run TextPipe from
the command line in a variety of modes
*
Show Command Line window - displays the command line parameters used to
invoke TextPipe - useful for debugging command line scripts
*
Check for TextPipe Updates window - check if you have the most recent
TextPipe by querying the Crystal Software web site
*
TextPipe now interfaces to 2 external tools (see Help\Related Products menu)-
  JustData from www.justsoft.com.au
  Offline Explorer from www.metaproducts.com

GUI enhancements
*
Tree view for filters which allows multi-level filters to be collapsed
*
Multiple-document interface (MDI) allows multiple filters to be open at once.
New Window menu to support MDI interface
*
Dockable toolbars
*
Icons in menus
*
Right-click (context) menus on all edit/memo fields allow special characters,
common patterns, replace expressions and macros to be inserted, as well as
the normal clipboard operations
*
Can now set the font used in the Ad-hoc Input/Output area

Bug fixes
*
ActiveX script now works in non-ad-hoc mode
*
Trial Run area now does not interpret \n, \t etc
*
Split files at Character - fixed bug

What's New in v4.6
==================
*
Example code for the TextPipe Engine - a text transformation .dll that can be
downloaded from www.crystalsoftware.com.au
*
New ActiveX scripting filter supports arbitrary user-written code in
VBScript, Jscript or PerlScript, PythonScript, REXXscript etc
*
Output To New Folder can now recreate multi-level folder trees and output to
a new drive
*
New Undo command to undo any changes to the current filter
*
New file grid button to Copy Row
*
New Extract filter to extract just the required text
*
Show Duplicates filter now called Count Duplicates, and outputs the number of
times found to the left of each item
*
New Tools menu item to pull sample data from the file list or clipboard into
the trial run area
*
New Tools menu item to move the Trial Run Output back to the Trial Run Input
*
Add Line Numbers filter now allows format of line numbers can be changed to
custom user-defined settings, including numbering in hexadecimal etc
*
Sort filter now has Ignore Case option
*
Header, footer, left margin or right margin can now insert either a file or
text. Cursor position now shown
*
Length of trial input and output area shown in both lines and characters.
Cursor position now shown
*
Warning indicator on status bar now shows if Test Run mode is active or not
*
Debug filter - creates a debug window showing the text passing through the
filter at that point
*
External filter files can now be linked to, providing true 'black box' style
re-use, using File\Link To Filter
*
Can now duplicate the current filter - this makes adding a large number of
similar filters very easy
*
Find and replace filters now use multi-line edit control - making it easier
to edit large selections. The new edit control also provides the length and
line count of the text
*
Maps now display a hex column in addition to the original Decimal and ASCII
columns
*
Changed handling of changing the replacement text for a pattern-based search
- now the default text is not modified for future replaces
*
Tips of the Day
*
Query replace window now supports resizing better
*
Changed registration URLs
*
The filter names shown in the filter list are now truncated, preventing
Windows crashes with very long filter names
*
Fixed logging functions which were going wild when more than 1000 files were
processed
*
Fixed pattern matching not handling \n \r \e \t \a
*
Fixed Randomize filter - was outputting blank text
*
Fixed pattern based search with patterns longer than 247 characters - was
truncating and matching early

What's New in v4.5
==================
*
Soundex search method to search for words that sound like another word.
Useful when you are replacing a typing error
*
New Preferences dialog
- Play sound on completion
- If no association, use user defined editor (need new options window)
*
Prepare filter for automation - removes all user prompting
*
Prepare filter for distribution - removes all test text
*
Can now insert a file with the Add Header, Add Footer, Add Left Margin and
Add Right Margin filters - browse button
*
Add Line Numbers now has option to skip blank lines, (ie omit count but keep
counting), as well as omit count and not count line)
*
Now can split files at a given character, after a specified number of
occurrences (and split before, after or on top of character)
*
ANSI to Unicode and Unicode to ANSI filters (single byte to double byte)
*
Comment filters now shown in italics, disabled filters shown with strike
through text, speed button to add comments
*
Better progress indicator - now much more accurate
*
Duration of filter job in seconds shown in status dialog
*
Increased size of edit fields for easier editing of multi-line text. Display
of text length and number of lines of all memo fields (like Header, Footer,
Left/Right Margin, Insert Column etc)
*
Performance improvement when handling very large files
*
Improved performance for Remove Blanks from Start of Line, Convert to Lower
Case, Convert to Upper Case, OEM to ANSI, ANSI to OEM
*
Query replace dialog now shows file currently being processed
*
Query replace dialog now shows filter information for both normal and pattern
based searches (so it is easy to see which text was matched by each
expression)
*
Improved command line automation support. Now can have comments embedded -
useful when converting Perl scripts to TextPipe scripts because code can be
commented out rather than edited out
*
Form resizing now handled better
*
Fixed bug with Query Replace when replacement text was modified before
replacing
*
Fixed bug with saving/loading of window positions
*
Fixed bug with detection of when the filter has been modified
*
Fixed bug with query replace 'Yes - file' button - did not change the first
entry

What's New in v4.4
==================
*
Name change from DropConvert to TextPipe
*
Regular expression search and replace - allows pattern matching (also known
as non-exact matches or regexps). Case-insensitive or case-sensitive matching
*
HTML Help integration
*
Prompt on replace. Can change the replacement text
*
Option to delete test files at end of test run
*
@inputFilename macro now allowed in filters for Add Text To Start Of Lines,
Add Text At End Of Lines, Add Column Of Text
*
New @clipboard macro to insert the clipboard contents
*
Can disable filters by double-clicking or pressing Ctrl+T
*
When either the error display or the file display is full it is sent to the
log file
*
/z command line parameter to read parameters from a file
*
Grep-based regular expression matching
*
In-place editing of file specifications
*
Inclusion and exclusion lists for file processing
*
File groups allow better handling of complex file groupings
*
Subfolder recursion can now be specified on a per-file basis. Improved
performance of subfolder recursion
*
Improved detection for binary files - sample size can be adjusted, and binary
characteristics are better identified
*
Case-sensitive replace - if found text is uppercase, replacement can be
forced to all uppercase etc
*
New Export Filter to clipboard feature. Useful for documenting filters in
Word
*
Enhanced detail when printing filters
*
File associations for .dCv test files
*
Status dialog now shows separate error tab, and shows files generated by each
filter, with the old and new size of each with a comparison display for quick
checks
*
Install/uninstall feature. Creates Start Menu items, desktop icon, Send To
menu and file associations for .fll and .map files and for text files
*
Scratch pad area for test text
*
Trial run area for iterative testing
*
Improved error log for command line initiation
*
Pasted text is scanned for binary characters - and they are converted to
readable text. Pasted text is now checked to see if it has already been
converted
*
Heaps of useful sample filters
*
Window sizes now saved and restored. Minimum window sizes enforced
*
File list now supports including and excluding files, file groups, subfolder
inclusion/exclusion on a per-line basis
*
Scratch Pad area for saving chunks of text
*
New Edit Menu with Cut, Copy, Paste and Disable/Enable Filter
*
Settings now saved on a per-user basis
*
Improved descriptions in filter window
*
Improved performance. Better use of memory (now uses far less)
*
Scrolling on drag of filters in filter list
*
More filters and options available from command line
*
Now handles filters from future versions better
*
Printer setup menu item now doesn't get lost :-)
*
Fixed bug with "Base map on filter list" function

What's New in v4.3
==================
*
Filter to insert a column of text
*
Filter to convert to ranDOm CasE
*
Filter to extract URLs. Very useful for retrieving mailto, http, https, ftp
and gopher addresses
*
Task bar menu. Allows filters to be loaded and run without leaving external
programs such as Word, enabling conversion to occur seamlessly without
leaving the currently running program
*
All status windows from completed tasks are closed when starting a new task
*
Input filter new options:
  Input from file or clipboard
*
Output filter new options:
  Output to file or clipboard (independent of Input filter)
  change file extension
  change base directory on output
*
Filter to only show duplicate lines
*
Filter to reverse each line
*
Filter to centre justify lines
*
Filter to right justify lines
*
Filter to remove lines contained in an external file
*
Filter to only show lines contained in an external file
*
Filter to run external program. Very useful for user-written filters or for
features not yet incorporated into DropConvert
*
Filter to add comments to the filter list
*
Character maths filter now shows hex and binary equivalents for operand
*
File history (up to 5 filters) shown in file menu
*
Tool bar buttons for commonly-used filters
*
Floating hints now used in preference to status bar
*
Explorer menus for .fll file type
*
Removed evaluation edition restriction of 5 filters
*
Now uses InstallShield install/uninstall


Web vs Standard vs Pro Feature Comparison

 Web vs Standard vs Pro Feature Comparison

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Please see the latest version of this chart online at

http://www.crystalsoftware.com.au/textpipe/pro_compare.html


Installation

 Installation

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
System requirements
Distribution Files

TextPipe is simple to install. Simply double-click on the .exe file that you downloaded (usually textpipepro.exe, textpipestandard.exe, textpipeweb.exe etc).

The installation program creates uninstall entries under Control Panel:Add/Remove Programs. These allow TextPipe to be cleanly removed from your system.

Uninstall

Uninstallation is completely automatic.

To uninstall TextPipe, click the Start Menu, Settings, Control Panel, Add/Remove Programs. Select TextPipe from the list presented, and click Add/Remove. TextPipe will then be deleted.

The uninstall does not remove files modified or added by you, so any changes you have made to filter files (.fll) or maps (.map) will not be lost.


Online Web Purchase

 Online Web Purchase

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Web vs Standard vs Pro Feature Comparison
Purchase Information

The commercial version of TextPipe can be purchased and received immediately on the Internet, eliminating shipping and handling costs. The latest price list can be downloaded from http://www.crystalsoftware.com.au/order.txt

Direct Purchase URL: http://www.crystalsoftware.com.au/buy.html

This purchase URL may be used to purchase all the following TextPipe licenses:

  • TextPipe Standard
  • TextPipe Web
  • TextPipe Pro
  • TextPipe 5 User Pack
  • TextPipe 10 User Pack
  • TextPipe Educational Site Edition
  • TextPipe Site Edition
  • TextPipe Server Edition
  • TextPipe Company Edition

See also

Purchase Information
 Order Form
License Information


CRYSTAL SOFTWARE ORDER FORM
===========================
Prices current as at Saturday, 1 January 2005
If you are using NotePad, please maximize the window before
printing.

All our products come with (unless stated below)
* One Year of FREE upgrades
* 30 days FREE Phone support
* Bonus additional software
* Unlimited Access to Technical Support web site and product FAQs
* Unlimited Access to Discussion Forums.

I N T E R N E T   O R D E R S
-----------------------------

Please go to

    http://www.crystalsoftware.com.au/buy

to pay by credit card using the Internet.

U P G R A D E S
---------------

One year of FREE upgrades is included with each product. The upgrade price varies for
each product - please see our website for upgrade price details:

    http://www.crystalsoftware.com.au/buy

P H O N E   O R D E R S
-----------------------

You can order by credit card over the phone if you do not have
Internet access.

         USA Office: +1 724-850-8186
         USA and Canadian customers: call 1-800-903-4152
         German Office: +49-221-31088-20

Please have the Product Id (see the Product list below) and
your credit card ready.

F A X / M A I L   O R D E R S
-----------------------------

You can order by fax or postal mail if you do not have Internet access. Please use
the order form below for:

    * Purchase orders (commercial, government, educational etc)
    * Fax orders
         * Postal mail orders
    * Cash payments
    * Check payments
    * All other non-internet orders

1. Please download the latest version (*) of this form from

    http://www.crystalsoftware.com.au/order.txt

    (*) Prices may not be honored if you do not use the
         latest version

2. Print out and complete this form

3. Fax or mail it to:

USA Office	German Office

ShareIt! Inc.	ShareIt!
PO Box 844	element 5 AG
Greensburg	Vogelsanger Str. 78
PA 15601-0844	50823 Köln/Cologne
U.S.A.	Germany

Tel. +1-724-850-8186	Tel: +49-221-31088-20
Fax. +1-724-850-8187	Fax: +49-221-31088-29

C O N T A C T   D E T A I L S
-----------------------------

 First Name: _________________________________________________

  Last Name: _________________________________________________

     E-mail: _________________________________________________

     ** Please do not forget to include your e-mail address **
        *** We will use e-mail to communicate with you. ***

    Company: _________________________________________________

    Address: _________________________________________________

             _________________________________________________

             _________________________________________________

             City: ___________________________________________

             State: __________________________________________

             Postal code: ____________________________________

             Country: ________________________________________

      Phone: _________________________________________________

        Fax: _________________________________________________

How would you like to receive the registration key/full version?
(Please tick one)

         [ ] E-mail (please ensure you have filled it in above)
         [ ] Fax (please ensure you have filled it in above)
         [ ] Postal mail (additional charge may be levied)

P A Y M E N T
-------------

How would you like to pay the registration fee? (Please tick
one)

         [ ] Credit card (please fill out the details below)
         [ ] Wire transfer
         [ ] EuroCheque
         [ ] Cash
         [ ] US Check drawn on a US bank

Credit card information (if applicable)
         [ ] Visa
         [ ] Eurocard/Mastercard
         [ ] American Express
         [ ] Diners Club
      [ ] Other: _________________________

Card Holder: ______________________________________________

Card Number: ____________________________ Expiry Date (mm/yyyy): ___/_____

Date / Signature: _________________________________________

P R O D U C T S
---------------

Please note: All prices are in USD. If you wish to order in Euros or another
currency, please indicate this when you order and will perform the appropriate
conversion.

** Please contact us for other licensing needs **

ID
DESCRIPTION AND PRICE
TextPipe Pro (fix text fast in web sites, database extracts and e-
text of unlimited size!)

Note: More features than Standard version. Please ensure
you are evaluating the Pro version before ordering

171179

131686
131686
131686
131822
136051
136049

__ x 10 Days Only License    @    $50: $______
(limit one per customer for 10 Days Only version)
1-2  Single User Licenses    @   $299: $______
3-9  Single User Licenses    @   $230: $______
10+  Single User Licenses    @   $220: $______
__ x Server License          @  $CALL: $______
__ x Site License            @  $CALL: $______
__ x Enterprise License      @  $CALL: $______

TextPipe Standard

Note: Less features than Pro version. Please ensure you
are evaluating the Standard version before ordering

137374
__ x Single User License     @   $169: $______

TextPipe Web

Note: Less features than Standard and Pro versions.
Please ensure you are evaluating the Web version before
ordering

157097
__ x Single User License     @    $40: $______

TextPipe Pro Run Time

Note: (the run time, read-only version of TextPipe Pro -
filters CANNOT be modified by clients)

147437
147438
__ x Single User License     @   $149: $______
__ x Server License          @   $249: $______

TextPipe for askSam (works only with askSam database)
168910
168911
__ x Single User License      @   $89: $______
__ x Run Time Single User     @   $39: $______

ResumePipe Subscription (convert emails and attachments to text
files) - includes TextPipe Pro filtering
Subscription based on number of resumes processed
200766
Resumes   C/Res   Total
  1,000   $0.25   $  250
  2,000   $0.22   $  440
  5,000   $0.19   $  950
 10,000   $0.16   $ 1600
 20,000   $0.13   $ 2600
 50,000   $0.10   $ 5000
100,000   $0.07   $ 7000
200,000   $0.05   $10000
500,000   $0.04   $20000
                                       $______

ResumePipe Unlimited Resume Model
137750
__ x Single User License     @  $6999: $______

DataPipe (interactive search and replace in databases)
138135
140629
155969

__ x Single User License     @   $590: $______
__ x Single CPU License      @   $590: $______
__ x Site License            @  $CALL: $______

WordPipe Subscription Model (search and replace in multiple Word
documents)
Subscription based on number of documents processed
200779
  Docs   C/Doc   Total
   100   $0.50   $  50
   200   $0.45   $  90
   500   $0.40   $ 200
  1000   $0.35   $ 350
  2000   $0.30   $ 600
  5000   $0.25   $1250
 10000   $0.20   $2000
 20000   $0.15   $3000
 50000   $0.10   $5000
100000   $0.06   $6000
                                       $______

WordPipe Unlimited Document Model
138134
__ x Single User License     @   $699: $______

WordPipe (Search Only) Unlimited Document Model
205364
__ x Single User License     @  $1299: $______

ExcelPipe Subscription Model (search and replace in multiple Excel
spreadsheets)
Subscription based on number of spreadsheets processed
203085
  Docs   C/Doc   Total
   100   $0.50   $  50
   200   $0.45   $  90
   500   $0.40   $ 200
  1000   $0.35   $ 350
  2000   $0.30   $ 600
  5000   $0.25   $1250
 10000   $0.20   $2000
 20000   $0.15   $3000
 50000   $0.10   $5000
100000   $0.06   $6000
                                       $______

ExcelPipe Unlimited Spreadsheets Model
186919

__ x Single User Licenses    @   $499: $______

TextPipe Engine Programmers DLL
132910

__ x per Application it is used in
     Royalty Free            @   $500: $______

AbusePipe spam complaint analyzer
188875

__ x Single User License     @  $3990: $______

EasyPattern Engine Programmers DLL
211244

__ x per Application it is used in
     Royalty Free            @   $199: $______

HookKeys Programmers DLL
TBA
__ x per Application it is used in
                             @   $CALL: $______

DetachPipe (save, detach and delete email attachments)
146668

138333
166349
151810
1-2  Single User Licenses    @   $129: $______
3-9  Single User Licenses    @    $85: $______
10+  Single User Licenses    @    $69: $______
__ x Home License            @    $35: $______
__ x Server License          @  $CALL: $______
__ x Enterprise License      @  $CALL: $______

EmailPipe (converts emails to text files)
198245
__ x Single User License     @  $499: $______

Time Zone Helper (show international working time overlap)
162196
__ x Single User License     @    $20: $______

DirSize disk space monitor
132558

140086
140426
1-4  Single User Licenses    @    $40: $______
5-9  Single User Licenses    @    $36: $______
10+  Single User Licenses    @    $34: $______
__ x Server License          @   $349: $______
__ x Site License            @  $CALL: $______

DirDate
132564

1-4  Single User Licenses    @    $20: $______
5-9  Single User Licenses    @    $18: $______
10+  Single User Licenses    @    $17: $______

Arc Menu (formerly Launch)
132561

157148
1-4  Single User Licenses    @    $20: $______
5-9  Single User Licenses    @    $18: $______
10+  Single User Licenses    @    $17: $______
__ x Enterprise License      @  $CALL: $______

ZeroIn
100503

1-4  Single User Licenses    @    $20: $______
5-9  Single User Licenses    @    $18: $______
10+  Single User Licenses    @    $17: $______

LFNit!
131928

1-4  Single User Licenses    @    $40: $______
5-9  Single User Licenses    @    $36: $______
10+  Single User Licenses    @    $34: $______

BabyShield keyboard and mouse disabler
131903
__ x Single User License     @  $9.99: $______

KeyboardLocker - let your child watch DVDs!
205713
__ x Single User License     @ $19.99: $______

Clean 'n' Go mouse, keyboard and monitor cleaner
195484
__ x Single User License     @  $9.99: $______

                                     ===================
                                     Total: USD
                                     ===================


Purchase Information

 Purchase Information

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

TextPipe is developed as a commercial product, and you may evaluate it to determine if it meets your needs. You must purchase TextPipe if you use it beyond the evaluation period or if you use any of the output for non-evaluation purposes. Use of TextPipe after the evaluation period is illegal and violates international copyright laws.

Benefits of Purchasing TextPipe

  • 60 days FREE email support
  • 60 days FREE updates
  • Access to user support groups
  • Notification of new releases, updates and upgrades
  • Access to pre-release products. You will receive access to the newest tools created by Crystal Software before they are released to the public.

Purchasing

There are two ways to purchase TextPipe:

  • The easiest, quickest and most convenient way is to visit the TextPipe home page on the Internet at http://www.crystalsoftware.com.au and use the online purchase facility. This facility is for credit card payment only.
  • Mail or email the order form to Crystal Software. The mailing address is contained in the form. The order form is contained in the help file and in ORDER.TXT
  • You can also order over the phone using the details in the order form

See also

Online Web Purchase
Order Form
 License Information


Other documentation

 Other documentation

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Contact Information
Consulting Services
Vendor Information
Corporate Modifications
Other Crystal Software Products

Contact Information
 Other Crystal Software Products
 Shareware Vendor Information
 Distribution Files
 License Information
 Corporate Modifications


Contact Information

For technical support, please see the support page.

For sales, please contact our sales department directly:

Envelope sales@crystalsoftware.com.au

For all other enquiries, please write to our general address:

Envelope mail@crystalsoftware.com.au

Phone/Fax

Phone - phonen1.gif (1029 bytes) +61-3 9888 3104 (9am-5pm Australian Eastern Standard Time - please see below). Within Australia please call (03) 9888 3104
Fax - Fax machine +61-3 8610 1234. Within Australia, please fax (03) 8610 1234.

Time Zones (GMT+10)

Here you can check if you are calling us in Australian business hours (9am - 5pm Monday to Friday)

Your Time | Your Date
Melbourne Time | Melbourne Date

Mail

Crystal Software
2 / 9 Woodbine Court
Wantirna
Victoria 3152
AUSTRALIA

Feedback and Comments

We're always open to feedback and comments! Please use our suggestion box to let us know if there is a feature you would like to see in future versions. We also welcome any reports of bugs or incompatibilities, inconsistencies, spelling mistakes, grammatical errors, broken links etc.


Other Crystal Software Products

Crystal Software develops system utilities for power users, administrators and programmers. It has developed 14 software titles since 1990:

More information about TextPipe TextPipe | Automated search and replace and text processing tool
More information about DataPipe DataPipe | Uses TextPipe Pro to modify database records live in your databases
More information about ResumePipe ResumePipe | Extract emails and their attachments from Outlook, Lotus Notes, Applix or MAPI-compliant email systems to sequentially numbered files, and post-process using TextPipe Pro
More information about DetachPipe DetachPipe | Save, detach and delete email attachments from Outlook, Lotus Notes, Applix or MAPI-compliant email systems
More information about WordPipe WordPipe | Perform multiple search/replaces in multiple MS-Word documents
More information about ExcelPipe ExcelPipe | Perform multiple search/replaces in multiple MS-Excel spreadsheets
More information about PowerPointPipe PowerPointPipe | Perform multiple search/replaces in multiple MS-PowerPoint presentations
More information about AbusePipe AbusePipe | Detect IP address of Spam offenders from email complaints
More information about EmailPipe EmailPipe | Convert emails to numbered text files
More information about Time Zone Helper Time Zone Helper | Displays the working time overlap for people in different time zones
More information about DirSize DirSize | Windows 95 command line utility to accurately show used and wasted disk space, sorted by total size, depth, wastage etc
More information about ZeroIn ZeroIn | Fast CD replacement with spelling correction, history, aliases and long filename support
More information about Arc Arc Menu | Text-based, mouse-driven windowed menu system with many advanced capabilities
More information about DirDate DirDate | Unique ability to change the date on folders
More information about BabyShield BabyShield | Temporarily disables the mouse AND keyboard while your pet or child taps on the keyboard
More information about ClipSize ClipSize | Displays memory occupied by the clipboard and allows it to be cleared easily
More information about Clean 'n' Go Clean 'n' Go | Temporarily disables the mouse, keyboard or screen while they are cleaned.
More information about LFNit! LFNit! | Restores Windows 95 long filenames. Designed to allow continued use of old 16 bit applications (eg XTree, tape backup software, LapLink, PkZip etc) without losing long filenames

Contact Information


Vendor Information

If you are a

  • Shareware distributor
  • Disk vendor
  • Web site
  • CDROM publisher
  • Magazine
  • or any other organization or person

you may distribute the Evaluation Editions (*) of all Crystal Software programs provided the following conditions are met:

  1. You must inform the buyer that they are paying for your service, not for the programs, and that the programs must be purchased if they wish to continue using them after the evaluation period.
  2. You must include all original files included with the program, and you must not modify these files in any way. If required, you may add new files of your own.
  3. You must let us know by email if you include our software in your collection. We can then keep you informed of the latest version.

(*) All programs use the Help\About dialog or the -? Command line parameter to identify that they are an Evaluation Edition.

Contact Information


Distribution Files

 Distribution Files

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

The TextPipe archive consists of the following files:

TEXTPIPE.EXE | The TextPipe program
*.FLL | Useful filters for common tasks
*.MAP | Useful maps for common tasks
DESC.SDI | Archive description file
FILE_ID.DIZ | Archive description file
ORDER.TXT | Order form
README.TXT | Readme file
TextPipe.HLP | Help file for 32-bit version
VENDOR.TXT | Vendor information
CORPORAT.TXT | Information about Corporate vesions
LICENSE.TXT | User license information
PRODUCTS.TXT | Information about other Crystal Software products
WHATSNEW.TXT | What's new in this version of TextPipe

If any of these files are missing from your copy of TextPipe then you can obtain a complete version from the Crystal Software Home Page on the Internet.


This program and the accompanying files and documentation are

Copyright (c) 1990- 2005 Crystal Software
All rights reserved

This file contains our end user license.  It also explains the terms of the
Commercial License where they're different from the Evaluation License.

If you have any questions, please contact us at

Address:
Crystal Software
2 / 9 Woodbine Court
Wantirna
Victoria 3152
AUSTRALIA
Phone:
(+61) 3 9888-3104 (GMT+10)
Fax:
(+61) 3 8610-1234
Web:
http://www.crystalsoftware.com.au

====================================================================
Plain English Evaluation License Agreement and Limited Warranty

1. SUMMARY.  You may install this SOFTWARE on as many computers as your license
allows, and no more:

You may access the software through a network, provided that you have
obtained individual licenses covering all computers that will run the
software from the network. For instance, if six different workstations will
access the software on the network, you will need a six-user license, whether
they use the software at different times or concurrently.

* Home License/Personal Edition
May be used by one person in a home environment. CANNOT be used in
government, business or any organization.

* Single User License/Desktop Edition/Single User Pack
May either be used by a single person who uses the software himself on one or
more non-server class computers, or installed on a single non-server class
computer and used by multiple people, but not both.

* 10 Days Only License (TextPipe Pro only)
May be used for 10 unique calendar days (consecutive or non-consecutive), as
determined by our license server.

* Single CPU License (per processor)
One license is required for each CPU in a computer. For example, if a
computer has four CPUs, four CPU Licenses are required.

* 5 User Pack
May be installed on UP TO FIVE (5) non-server class computers ONLY

* 10 User Pack
May be installed on UP TO TEN (10) non-server class computers ONLY

* Server License
May be installed on ONE (1) server class computer ONLY

* Site License/Educational Site License
May be installed on UP TO 500 computers of server class or non-server class
at one site ONLY

* Enterprise License/Company Edition/University Edition
May be installed on UP TO 2000 computers of server class or non-server class
directly owned by a Company or University ONLY (but not on client's
computers)

This is a legal agreement between you and CRYSTAL.  By using the SOFTWARE you are
agreeing to be bound by the terms of this Agreement.  Do not use the SOFTWARE if
you do not agree to the terms of this Agreement.

2. DEFINITIONS.  In this agreement:

*
"SOFTWARE" means the computer program you have purchased or obtained
for evaluation, including the program, associated documentation and
other materials;

*
"Evaluation Copy" means a copy of the SOFTWARE distributed by us or by
our authorised agents for evaluation purposes and which is described
as an "Evaluation Edition" or a "Shareware Edition" by the SOFTWARE's
"About" screen;

*
"Commercial Copy" means a copy of the SOFTWARE purchased from us or
from a dealer, and is NOT described an an Evaluation Copy by the
SOFTWARE's "About" screen.  The actual description may include
"Desktop Edition", "Single User Edition", "Personal Edition", "Server
Edition", "Site Edition" or "Company Edition" or other variations
depending on the version purchased;

*
"you" means the end user of the SOFTWARE.  This can be either an
individual or an entity;

*
"server class" means a computer designated as a file server, database
server, application server or other kind of server;
"desktop class" or "non-server class" means a computer where only one
person uses it at a time;

*
"CRYSTAL" means Crystal Software, an Australian corporation (address
shown above).

3. GRANT OF LICENSE.  CRYSTAL grants you the right to INSTALL this software on as
many computers as you have purchased a license for.  You may NOT install it on
computers that you do not have a license for.

The SOFTWARE is "installed" on the computer when it is loaded into permanent memory
(e.g. hard disk, CD-ROM, or other storage device) of the computer.

4. COPYRIGHT.  The SOFTWARE is owned by CRYSTAL and is protected by copyright laws
and international treaty provisions and all other applicable national laws.
Therefore, you must treat the SOFTWARE like any other copyrighted material (e.g. a
book or musical recording) except that you may make one copy of the SOFTWARE solely
for backup or archival purposes.  You may not copy the user documentation provided
with the SOFTWARE.

5. OTHER RESTRICTIONS.  You may not rent, lease, sell, or attempt to rent, lease or
sell the SOFTWARE. You may transfer the SOFTWARE and accompanying written materials
on a permanent basis provided you retain no copies and the recipient agrees to the
terms of this Agreement.

You must not copy, translate, adapt, modify, alter, decompile, disassemble, or
otherwise reverse engineer or create any derivative work of the SOFTWARE, merge the
SOFTWARE or any part thereof with any other software or change any SOFTWARE in
whole or in part, nor allow any other party to do so.  You must not alter or remove
any copyright or other intellectual property notifications applied to the SOFTWARE.

You acknowledge that the SOFTWARE and the copyright, and other intellectual
property rights of whatever nature in the SOFTWARE are not your property and
contain valuable proprietary information and trade secrets and that this license
confers no title or ownership in the SOFTWARE and no rights in any associated
source code, and is not a sale of any rights in SOFTWARE.

6. EVALUATION PERIOD.  You may use an Evaluation Copy of the SOFTWARE for up to 30
days in order to determine whether it meets your needs.  While you are evaluating
the SOFTWARE, you may use it on as many computers as are required to perform your
evaluation.  Your evaluation period begins when you first install the SOFTWARE on
one or more computers for evaluation purposes.  Once the evaluation period is over
you MUST PURCHASE THE SOFTWARE or REMOVE IT FROM ALL COMPUTERS. If you use the
software for commercial purposes DURING your evaluation you MUST PURCHASE THE
SOFTWARE.

7. MAKING AND DISTRIBUTING EVALUATION COPIES OF THE SOFTWARE.  You may make copies
of your Evaluation Copy of the SOFTWARE to give to others, as long as you include
all of the files that you originally received with your Evaluation Copy.  When you
give an Evaluation Copy of the SOFTWARE to another person, you agree to inform them
that their copy is to be used for a time-limited evaluation period, and that they
must purchase a Commercial Copy if they continue to use the SOFTWARE once the
evaluation period has ended.  You agree not to sell Evaluation Copies of the
SOFTWARE or distribute them to others for any kind of compensation or fee unless
the fee solely covers your costs in providing it to the end user.

8. OPERATION OF THE SOFTWARE.  The SOFTWARE offers powerful capabilities.  If you
do not follow our instructions, or if you use the SOFTWARE improperly, you can
destroy files or cause other damage to your software and data.  You assume full
responsibility for the selection and use of the SOFTWARE to achieve your intended
results.

9. LIMITED WARRANTY.  The following limited warranty applies to Commercial Copies
of the SOFTWARE.  It is included here so you understand what your warranty will be
when you purchase a Commercial Copy. This warranty does NOT apply to Evaluation
Copies of the SOFTWARE.

Bugs and SOFTWARE Errors. We don't promise that the SOFTWARE will be free of bugs
or errors. If you report an error or bug to us, we will use our best efforts to
correct it.

10. DISCLAIMERS AND LIMITATIONS.  And now the legal language we couldn't avoid:

DISCLAIMER FOR EVALUATION COPIES: YOU ARE NOT PAYING FOR THE TRIAL USE LICENSE FOR
YOUR EVALUATION COPY OF THE SOFTWARE.  ACCORDINGLY THE EVALUATION COPY IS PROVIDED
"AS-IS" AND WITHOUT WARRANTY OF ANY KIND.

DISCLAIMER FOR COMMERCIAL COPIES: Except for the limited warranty stated above, we
disclaim any and all other warranties, express or implied, oral or written,
including any implied warranties of merchantability or fitness for a particular
purpose.  The limited warranty above gives you specific legal rights, but you may
have other rights, depending on where you purchased the SOFTWARE.

LIMITATION OF LIABILITY.  IN NO EVENT SHALL WE BE LIABLE FOR ANY INCIDENTAL,
CONSEQUENTIAL, OR PUNITIVE DAMAGES WHATSOEVER ARISING OUT OF USE OF THE SOFTWARE OR
YOUR RELATIONSHIP WITH US.  SOME COUNTRIES OR STATES DO NOT ALLOW EXCLUSION OR
LIMITATION OF LIABILITY FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES, THEREFORE THE
ABOVE LIMITATION MAY NOT APPLY TO YOU.

11. This agreement is to be interpreted under the laws of the state of Victoria,
Australia.

END OF LICENSE AGREEMENT


Corporate Modifications

Corporate and business users are welcome to approach Crystal Software for modifications specific to their needs. We can easily incorporate any changes you desire. Please send us specifications and we will send you a quote. Please also include details such as a contact name, phone and/or address, plus any other details you think are relevant.

Contact Information


Lesson 1 - Basics

 Lesson 1 - Basics

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

In this lesson you'll learn about:

  • What is TextPipe?
  • Opening TextPipe
  • The main elements of TextPipe's screen
  • How to use the Trial Run area
  • How to open (load) and save filters.

What is TextPipe?

TextPipe is a powerful text transformation workbench that performs an enormous range of conversions and manipulations to text files. It is unique in the number of transformations that can be performed at one time, in the power and flexibility of its transformations, in the size of files it can handle and more. Data can be extracted from databases in a variety of text formats, and custom transformations can be written in VBScript, JScript and other scripting languages.

Opening TextPipe

If you haven't already started TextPipe, start it now by

  1. Double-clicking on the TextPipe icon textpipe.gif (1071 bytes) on the desktop OR
  2. Clicking on TextPipe in the Start Menu\Programs\TextPipe folder

TextPipe starts with a new filter list (you may have to click Ok to clear the About dialog in the evaluation version, and close the Tip Of The Day and Filter Wizard dialogs).

The TextPipe Screen

At the very top of the TextPipe window is the Title Bar. The Title Bar tells you which filter list you are working on and whether it has been modified or not. Below the Title Bar is the Menu Bar, with the customary File, Edit, Window and Help menus, along with a Filter menu that categorize text processing filters into functional groups (we'll come back to filters in a minute).

Below the Menu Bar is the Tool Bar. The Tool Bar has icons for the most commonly used features of TextPipe, such as saving and loading filter list files, cutting and pasting and so on. The Tool Bar saves you having to navigate through the menus. If you hover the mouse cursor over a Tool Bar icon for a second, a hint will be displayed beside the mouse cursor that describes what the icon is for. At the same time, a helpful message will be displayed in the Status Bar at the base of TextPipe's window. You should watch for helpful text in the Status Bar whenever the mouse cursor is over a field in TextPipe's window, because the Status Bar always has extra information for you.

The Tab Bar is below the Tool Bar, and it contains a group of tabs that divide up the main tasks within TextPipe. We'll cover it in more detail below.

What is a filter list?

A TextPipe filter list consists of:

  • A list of operations to perform (filters)
  • A list of files that will be processed
  • (Optionally) Trial text that can be used to test the filter list.

To understand this more fully, we'll open a sample filter.

Opening (loading) a filter list

  1. On the File Menu, click Open
  2. The Open Filter List window appears
  3. Double click on the Tutorial folder (if you can't find it, it's usually in C:\Program Files\TextPipe\Tutorial). Note - you can load any filter from the subject area subfolders such as coding, conversions, email, etext, fun, html, sysadmin or vbscript
  4. Double click on 'lesson1'
  5. TextPipe opens the lesson1 filter list.

The default tab is the Filters to Apply tab. It shows which filters will be performed, and in what order. Order is very important! In our lesson1 filter list, the only filter we will use is a straight-forward search and replace filter, which replaces capital T with lowercase T. We will cover the File Input and File Output filters later.

Click on the Files to Process tab. Although currently empty, this is where we list all the files that should be processed.

Click on the Trial Run tab. The Trial Run Input area already has some sample text. When we test the filter with a Trial Run, the Trial Run output appears on the right hand side so that we can compare the output with the original, and make sure that the filter works as we expect. This can be really handy for complex filters.

Click the Trial Run button at the base of TextPipe's window. Some new text appears on the right hand side - with all capital T's replaced with lowercase t's (3 in all). This is exactly what we expect - because we have a single filter that is called Replace [T] with [t].

Add the text 'This is my line of text.' to the end of the text in the Trial Input area, and click the Trial Run button again. The new output text is shown.

Click the Trial Run button again. TextPipe toggles between the Filters to Apply tab and the Trial Run tab each time you click this button, although if you've changed the Trial Input text (as we just did), it will re-process the text and display the output in the Trial Run Output area without toggling.

Saving a filter list

If you want to save the filter list:

  1. On the File Menu, click Save.

If you want to save the filter list with a new name:

  1. On the File Menu, click Save As
  2. The Save Filter List window appears
  3. Type in a new name
  4. Click the Save button.

In the next lesson we'll look at

  • How to add filters to the filter list
  • The importance of filter order.

Exercise 1

  1. Open one of the filters in TextPipe's 'fun' folder - try Elmer Fudd.fll (usually C:\Program Files\TextPipe\Fun)
  2. Type some text of your own into the Trial Run Input area
  3. Click the Trial Run button to see what they do.

Tutorial Index


Lesson 2 - Changing the Filter List

 Lesson 2 - Changing the Filter List

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

In the last lesson we covered

  • The main elements of TextPipe's screen
  • How to use the Trial Run area
  • How to open and save filters.

In this lesson we'll look at

  • How to add filters to the filter list
  • The importance of filter order.

Before we begin, open the filter called 'tutorial\lesson2.fll'

Adding Filters to the Filter List

New filters can be added to the filter list to perform additional processing steps. The Filter menu group filters by function. To add a new filter, select a menu item from the Filter menus and it will be added, below the currently selected filter.

Filters cannot be added or moved above the File input filter or below the File output filter. It doesn't make sense to put filters before or after these items.

To demonstrate adding new filters, we will add to the filter list we started with in Lesson 1. Let's add a simple search and replace filter - but this time, replacing capital 'T' and replacing with '*':

  1. Select the Replace filter (replacing 'T' with 't')
  2. Select Filters Menu\Replace Menu\Find exactly
  3. A Replace filter is added below the current filter
  4. Type the capital letter 'T' (without quotes) in the Find section
  5. Type the letter '*' (without quotes) in the Replace With section
  6. Ensure the Match Case check box is checked
  7. Click the Trial Run button.

The text will be unchanged from before - the addition of the new filter has not made any difference.

Why? Order is important!

Filters are always applied from top to bottom, corresponding to the way they are shown.

The order that filters are applied is VERY important to TextPipe. Let's see why.

  1. Switch back to the Filters to Apply tab (remember - you can click the Trial Run button to toggle back)
  2. Select the Replace [T] with [*] filter
  3. Click the Up arrow button at the bottom of the window, just to the right of the [Remove All] button
  4. The Replace [T] with [*] filter is now moved up one position
  5. Click the Trial Run button.

Now the changes are made correctly. The second filter is unnecessary because all capital T's have already been replaced.

An important part of advanced use of TextPipe is understanding how to break a task down into smaller steps. There are thousands of uses for TextPipe's filtering by just combining filters together.

In the next lesson we'll look at

  • How to remove filters from the filter list
  • How to move filters in the filter list
  • How to disable filters.

Exercise 2

Position the mouse cursor over each icon in TextPipe's toolbar (at the top of TextPipe's window), and read the hint that appears in TextPipe's status bar (at the base of TextPipe's window).

Hints are also available on every control in TextPipe's windows.

Tutorial Index


Lesson 3 - Moving and deleting Filters

 Lesson 3 - Moving and deleting Filters

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

In the last lesson we covered

  • How to add filters to the filter list
  • The importance of filter order.

In this lesson we'll look at

  • How to move filters in the filter list
  • How to remove filters from the filter list
  • How to disable filters.

Moving Filters in the Filter List

In the previous lesson we briefly touched on how to move filters in the filter list.

To move a filter up or down:

  1. Select the filter
  2. Drag it up or down to its new position.

Deleting Filters from the Filter List

To delete a filter from the filter list:

  1. Select the filter
  2. Click the 'Remove' button in the lower left hand corner of TextPipe's window or press the 'Delete' key on the keyboard
  3. The filter is removed.

Disabling Filters

To disable a filter:

  1. Select the filter
  2. Double click it, OR right click it and uncheck the 'Enabled' flag, OR select Toggle filter Enable/Disable from the Edit Menu.

In the next lesson we'll look at

  • Adding files.

Exercise 3

  1. Load the filter file called 'tutorial\Lesson3.fll'
  2. Delete the filter(s) that are shown with delete in their title (ignore the plus signs next to the filter titles)
  3. Move the filter(s) to the location shown in their title
  4. Disable the filter(s) shown with disable in their title
  5. Click the 'Trial Run' button
  6. If you've done everything right, the Trial Run output should now display

    You've successfully completed
    exercise 3!    '

Tutorial Index


Lesson 4 - Adding files

 Lesson 4 - Adding files

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

In the last lesson we covered

  • How to move filters in the filter list
  • How to remove filters from the filter list
  • How to disable filters.

In this lesson we'll look at

  • Adding files.

Before we begin, open the filter called 'tutorial\lesson4.fll'

Adding files

(Note - if you're using TextPipe for askSam, you won't be able to add files. Please try the askSam tutorial online, or go to the next lesson).

So far we've just been processing text that's held in TextPipe's Trial Run Area. Now we want to get it to work on some real files, so let's see exactly how we do this:

  1. Open Windows Explorer
  2. Display the folder 'C:\Program Files\TextPipe'
  3. Locate the file called 'tips.txt'
  4. Drag and drop the file from Explorer into TextPipe's filter window (with the title 'Untitled..'). If TextPipe's window is not currently visible, drag (and hold the button down) the file to the task bar area where it says 'TextPipe', pause for 2 seconds until TextPipe comes to the foreground, and then drop the file into TextPipe's window
  5. TextPipe changes to show the Files to Process tab, with the new file added

It's that easy to add a file or files - just drag and drop them from Explorer.

To process the file:

  1. Click the Go button
  2. TextPipe displays the status window
  3. TextPipe processes the file, and then plays a sound to let you know it has finished
  4. Click on the Results tab
  5. Find the line in TextPipe's window (not in the Help) that says 'Output file [C:\Program Files\TextPipe\tips.txt] Size: ...'
  6. Double click this line
  7. TextPipe opens NotePad to show you the output file.

You can see in the output file that all spaces have been converted to underscore characters '_'. This is what the lesson4.fll filter file is designed to do.

In the next lesson we'll look at

  • Sub filters (filter cascades)

Exercise 4

  1. Modify the lesson4.fll file to replace the underscore characters '_' with spaces, and change tips.txt back to how it was originally.
  2. Open the output file and make sure it has spaces in it.

Tutorial Index


Lesson 5 - Restrictions and sub filters

 Lesson 5 - Restrictions and sub filters

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

In the last lesson we covered

  • Adding files.

In this lesson we'll look at

  • Restrictions and sub filters

What is a restriction?

Put simply, a restriction filter allows only a limited section of text to be processed. It does this using sub filters, which are shown below it and indented to the right in the filter view. A restriction filter to delete 11 lines starting at line 3 might look like this:

A restriction might be lines 100 to 199, columns 30 to 36, lines containing the text 'ERROR', a range of CSV or Tab-delimited fields or a combination of these subsets, such as columns 3 to 17 of lines 20 to 55. Restrictions and sub filters are very handy when you only want to apply a filter to a subset of the text, such as only performing a search and replace within the first 10 lines. Many other variations are possible.

Click here to find out how to add a sub filter

Display of sub filters

TextPipe displays sub filters with a filter hierarchy or filter tree, very similar to the way that Windows Explorer shows drives A: and C: as subsets of My Computer.

When you see a hierarchy of filters in TextPipe you know that a subset of the text is being treated specially. You can imagine the very top part of TextPipe's filter display to be selecting a very large subset - the entire file.

Creating a sub filter

To create a sub filter, you first need to Select the subset of text that you want to treat specially. You do this by choosing one of the filters in the Restrict Menu.

Let's do this now. We will create a filter that will replace the word 'TextPipe' with 'TextPipe Pro' but only in lines 1 to 3:

  1. Open the filter called 'tutorial\lesson5.fll'
  2. From the Restrict Menu, choose Line Range
  3. Customize the Line Range filter so that it will 'Select lines 4..7'
  4. From the Replace Menu, add a Find exactly filter
  5. Customize the Find and Replace filter so that it will find the text 'TextPipe' and replace it with '*TextPipe Pro*'
  6. Drag and drop the Find and Replace filter so that it appears cascaded beneath the Line Range filter. To do this, you need to drag the Find and Replace filter to the right hand side of the tree view before you drop it (or ensure that you are holding down the Shift key). As you do this, the Status Bar will display the text 'Valid Action:Dropping Source filter as sub filter of Target filter'
  7. Click the 'Trial Run' button.

After the Trial run completes, you'll notice that only the second and third occurrences of 'TextPipe' have been replaced with 'TextPipe Pro'. This is exactly what we wanted - the Line Range sub filter has only passed lines 4 to 7 onto the Search and Replace sub filter. We can confirm this easily by adding an UPPERCASE filter after the Find and Replace filter.

  1. Select the Find and Replace filter
  2. From the Conversion Menu, add an UPPERCASE filter
  3. The UPPERCASE filter is added immediately after the Find and Replace filter
  4. Click the 'Trial Run' button.

With this example we can see that multiple filters can be applied within a sub filter.

Expanding and Collapsing the Filter Tree

Just as you would with Windows Explorer, you can single click the plus sign [+] next to a filter item to expand it out and show its sub filters, and you can single click the minus sign [-] next to a filter item to contract it and hide its sub filters.

Comments

Comment filters can be used to document sub filters inside them. This is handy because you can hide the details of an operation inside a comment filter, and then collapse the comment filter so that the implementation is hidden.

Exercise 5

  1. Create a sub filter based on the example above that selects Columns 1 to 35 of lines 1 to 4, and replaces 'TextPipe' with 'TextPipe Pro' within that selection. Only one replacement should be performed.

Tutorial Index


How To Use TextPipe

 How To Use TextPipe

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
File Lists
Filter Lists
Comments Tab
Scratch Pad Tab
Trial Run tab
File Menu
Edit Menu
Tools Menu
Window Menu
Help Menu
Status Tab
Results Tab
Advanced Topics
Popup menus
Sub filters

Using TextPipe is simple:

  1. Choose the filters you wish to apply.

    Filter can be chosen from the Convert, Add, Remove, Email, Special and Map menus. Most filters can be further customized in the right hand area of the screen - just click on the filter you wish to customize.

  2. Choose the files you wish to process (or select the clipboard).

    You can drag and drop files from Explorer onto TextPipe's window (anywhere) and they will be placed in the file list. Folders dragged will have "*.*" added to their name - meaning all files in the folder will be processed. You can process the clipboard if you select the Input filter (at the very top of the filter list) and then select Input from Clipboard.

  3. Click the "Go" button to start conversion.

    If you wish to override the Input/Output filter settings and instead, take input from the clipboard and then send output to it, click the "Clipboard" button.

While the conversion is in progress, you can click the "Stop" button to cancel it, or you can use the main window to build up another conversion job.

The Clipboard button is always enabled, even if the clipboard is empty. The Go button is only enabled if the file list contains at least one file, or if the input filter specifies that Input comes from the Clipboard.

See also

Drag and Drop
 File Lists
 Filter Lists


Filter Lists

 Filter Lists

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up
Input Filter
Output Filter
Conversion Filters
Unicode Filters
Add Filters
Remove Filters
Email Filters
Search and replace
Special Filters
Restriction Filters
Map Filters

The Filters to Apply tab of the main TextPipe window controls the operations that are performed on each file.

The left hand portion of the Filters to Apply tab shows a brief description of each filter. Selecting a filter in the tree view displays the configurable parameters for that filter on the right hand side. In the example above, the Output Filter is selected in the list, and the right hand portion of the window shows the settings that apply to the Output Filter.

Double clicking on a filter toggles it between enabled and disabled state. A filter in that has been disabled does not take part in filtering operations, and its icon is shown with a cross through it.

Adding filters

To add a new filter to the list:

  1. Place the cursor ABOVE the position where you would like the new filter. The new filter will be added below the cursor
  2. Choose a filter to add from the Filter Menus (Convert, Add, Remove, Email, Special, Select, Maps)
  3. The new filter will be added below the cursor
  4. Edit the filter's properties on the right hand side of the window

Remove

To remove a filter, select it and press [Delete] or click [Remove].

Remove All

To remove all filters, click [Remove All], which will then ask for confirmation.

Drag and Drop

Filters can be dragged and dropped within the filter list to change their position. Filters cannot be dragged to a position prior to the Input Filter or after the Output Filter. During drag and drop, moving the mouse above or below the list causes the list to scroll. The further the mouse is away from the list, the faster it will scroll.

To create a sub filter, drag a filter onto another filter that supports sub filter (the mouse cursor will display a barred circle if the filter does not accept sub filters). You must position the mouse cursor in the right hand side of the tree view to create a sub filter. An alternative to positioning the mouse in the right hand side of the tree view is to hold down the [Shift] key while you drag. You can also hold down the [Alt] key when you add a new filter and it will be added as a sub filter of the current filter (if allowed).

Watch the status bar for information of whether TextPipe is going to simply swap filter positions or if it will create a sub filter.

Right-click menu

The tree view has a context menu which can be used to Enable or Disable a filter, delete it, Cut, Copy or Paste a filter or group of filters, and perform partial trial runs.

Partial Trial Runs

You can perform a Trial Run using the text in the Trial Run area. This is initiated using the [Trial Run] button. You can also perform a partial trial run by right clicking on a filter, and choosing a trial run of the selected filter or all filters up to that point. This is a very handy way of debugging filters.

Keyboard commands

Alt+Up | Moves the selected filter up one position
Alt+Down | Moves the selected filter down one position
Alt+(add a new filter) | If allowed, TextPipe adds the new filter as a sub filter of the currently selected filter.
Shift+(add a new filter) | TextPipe adds the new filter at the parent level of the current filter.
Shift+(drag a filter) | Tries to add the dragged filter as a sub filter regardless of whether the mouse is in the right hand half of the tree view or not.

Command Line

A filter list file can be opened on the command line by using the /F=<filename> option.

See also

Drag and Drop
 File Lists
Sub filters


Sub filters

 Sub filters

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Sub filters allow a restricted part of the entire text to be operated on by a group of filters without effecting the entire text. For example, a Restrict to delimited fields (CSV, Tab, Pipe etc) filter can pick out a range of CSV fields, and then a search/replace filter can operate JUST on the text restricted.

Creating a sub filter

To create a sub filter, drag a filter onto another filter that supports sub filter (the mouse cursor will display a barred circle if the filter does not accept sub filters). You must position the mouse cursor in the right hand side of the tree view to create a sub filter. An alternative to positioning the mouse in the right hand side of the tree view is to hold down the [Shift] key while you drag. You can also hold down the [Alt] key when you add a new filter and it will be added as a sub filter of the current filter (if allowed).

Watch the status bar for information of whether TextPipe is going to simply swap filter positions or if it will create a sub filter.

Keyboard commands

Alt+Up | Moves the selected filter up one position
Alt+Down | Moves the selected filter down one position
Alt+(add a new filter) | If allowed, TextPipe adds the new filter as a sub filter of the currently selected filter.
Shift+(add a new filter) | TextPipe adds the new filter at the parent level of the current filter.
Shift+(drag a filter) | Tries to add the dragged filter as a sub filter regardless of whether the mouse is in the right hand half of the tree view or not.

Other examples

Delete 11 lines starting at line 3

Hex encode tab fields 3 to 7

Many different filters support sub filters.

Note - sub filters treat each section of restricted text as a complete file.

See also

Restrict to delimited fields (CSV, Tab, Pipe etc)
Restrict line range
 Restrict column range
More restriction filters
Search/replace
 Add left margin
 Add right margin
 Add header
 Add footer
 Insert column
 Comments - not really a sub filter, but can be used to group filters together


File Lists

 File Lists

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up
Drag and Drop

Files to Process Tab

The Files to Process tab of the main TextPipe window allows you to configure the list of files to be processed. This list is only used if the Input Filter is set to process files.

Note: This screen shot shows an older version of TextPipe where Add=Include and Subtract=Exclude.

The columns widths can be resized by dragging the column separator in the header.

Filename or Folder Name Column

The Filename column lists both individual filenames and wildcard specifications for files to be matched. e.g. c:\windows\*.ini will match all files with an extension of .ini in the c:\windows folder. If the filename doesn't contain wildcards (* or ?), then double clicking it or clicking the [View] button when the row is selected will open the file in its associated application. Single clicking the filename after it has been selected allows it to be edited, and also shows an ellipsis button, which when clicked, allows you to browse for files.

Multiple file specifications can be separated with a semi-colon e.g.

c:\INetPub\*.htm;*.txt;*.asp

Clicking on the Filename column heading sorts the file list. Double clicking on a row in the file list opens the file or folder.

Note - TextPipe allows files to be specified more than once. This can be useful when a file needs to be appended multiple times.

Subfolders Column

The setting in the Subfolders column determines whether TextPipe scans subfolders below the given location. For example, if the filename column contained c:\Program Files\*.txt, then setting the Subfolders column to None will only process *.txt files in this folder alone. Setting the Subfolders column to All will process all *.txt files in every subfolder below c:\Program Files, such as c:\Program Files\Microsoft Office, c:\Program Files\Microsoft Office\Templates etc. Single clicking the cell allows it to be edited, where it provides a drop down list with values All, None, 1 level, 2 levels etc.

Action Column

The setting in the Action column determines whether TextPipe adds this file specification to the list or removes files that match it from the list. For example (as above), if the file list contained an entry to include *.txt in all subfolders, and below it an entry to Exclude all entries matching 'Readme.txt' from all subfolders, then TextPipe will look at the file list it has gathered so far, and remove all matching entries eg.

Sample File List After 'Include *.txt in all subfolders'

d:\test.txt
d:\Readme.txt
d:\winnt\readme.txt
d:\winnt\modemDet.txt
d:\winnt\setuplog.txt

Sample File List After 'Exclude Readme.txt in all subfolders'

d:\test.txt
d:\winnt\modemDet.txt
d:\winnt\setuplog.txt

Exclude entries are shown in Red. The Action column also provides the value Ignore which may be used to temporarily ignore any entry in the list.

Right click - Copy to Trial Input

Right clicking on a file shows a menu - allowing you to copy the selected file into the Trial Input area. If the file you click on has wildcards then TextPipe will pick the first file that matches the wildcards. Note that TextPipe ignores the settings of the Subfolders and Action columns.

Analyse

Attempts to detect the file encoding. Encodings detected include:

Unicode encoding UTF-32BE
Unicode encoding UTF-32LE
Unicode encoding UTF-16BE
Unicode encoding UTF-16LE
Unicode encoding UTF-16BE XML
Unicode encoding UTF-16LE XML
Unicode encoding UTF-8 XML
Unicode encoding UTF-8
EBCDIC XML
Unicode UTF-32BE
Unicode UTF-32LE
Unicode UTF-16BE
Unicode UTF-16LE
Unknown - ASCII or ANSI

Hex Dump

Displays a hex dump of the first 200 characters of the file. This is very handy for determining the file format.

Add Folder

Clicking the [Add folder] button displays a browse folder dialog, which lets you select a folder to be processed. When the folder is added, by default, '*.*' is appended to the folder name to indicate that all files in the folder will be processed. The new row containing the added folder defaults its Subfolder and Action columns from the row above.

Insert Row

Inserts a blank new row below the current row.

Copy Row

Inserts a new row below the current row that is an exact copy of the current row.

View

If the selected row's filename doesn't contain wildcards (* or ?), then double clicking it or clicking the [View] button when the row is selected will open the file in its associated application.

New Group

Adds a new file group below the current row. A file group limits the entries affected by Exclude entries. Exclude entries only act on files previously added within the current group, rather than affecting all files added so far.

Groups must have a colon ':' as the first character of their name, but the name is otherwise unrestricted.

Open File List

TextPipe can read the file list from a file. This means that you can easily create a list of files to process for various tasks, and open it when required rather than go through the laborious task of adding the files to the list each time.

The file consists of a single file specification per line eg.

c:\my documents\*.txt
c:\bp\*.pas
d:\job_ads\*.*

To open a file list, click on the open icon on the files tab. An open dialog box appears. Select one or more files to open and click on the Ok button.

Save File List

The currently displayed file list can be saved to a file by clicking on the Save icon on the Files tab. A save dialog appears and an existing filename can be selected or a new one specified.

Remove

To delete one or more entries in the list, select them and and press [Delete] or click [Remove].

Remove All

To remove all entries, click [Remove All]. You must confirm this action.

Drag and Drop

Drag and drop files from Windows Explorer either onto TextPipe's main window or onto its icon (the Files tab does not have to be visible).

Command Line

Provide the names of the files you wish to process on the command line, either by name (using wildcards) or in a list file. A file list can be opened on the command line by using the /F=<filename> option.

See also

Drag and Drop
 Filter Lists


Comments Tab

 Comments Tab

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

The Comments Tab is no longer used. Instead, comments may be placed freely throughout the filter list. Any old comments found when a filter is loaded are converted to a comment filter at the top of the filter list.

See also

Drag and Drop
 Filter Lists


Scratch Pad Tab

 Scratch Pad Tab

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

The Scratch Pad Tab is no longer used. If you load a filter that had Scratch Pad text, the tab is displayed as before. In future, users are expected to save their temporary text in notepad, using the convenient notepad icon in the Tools menu.

See also

Drag and Drop
 Trial Run Tab
 Filter Lists


Trial Run tab

 Trial Run tab

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

The Trial Run tab of the main TextPipe window gives you a test bed for trying out filters.

The Trial Run area is used to test a filter repeatedly on small sets of text. The Trial Run input text is saved with each filter.

When the Trial Run Tab is NOT displayed, clicking the Trial Run button displays the Trial Run tab, re-runs the current filter over the input text and shows the result in the output text area. Clicking the Trial Run button while the Trial Run tab is displayed switches the display to the Filters tab.

Note: The Trial Input area does NOT interpret any special character codes or macros.

Note 2: The Trial Run function can give slightly different results to those encountered when actually processing a file. The difference is because the Trial Input field automatically converts line feed characters into DOS line feed characters (#13#10). This means that while text in the Trial Input field may process correctly, the results from processing a file may be different if you are processing files with Unix or Macintosh line feed characters. The best way to avoid these problems is to include a convert EOL characters filter at the very start of your filter list, converting from Auto to DOS.

Extra Options

Treat Trial Input as Unicode

When checked, the text in the Trial Input area is filtered as Unicode (i.e. double-byte UTF-16LE). When this flag is unchecked, Unicode text is first converted to ANSI text before it is processed. Normally when this setting is checked, Treat Trial Output as Unicode is also checked.

Treat Trial Output as Unicode

When checked, the text sent to the Trial Output area is assumed to be Unicode (i.e. double-byte UTF-16LE). Only use this setting if

  1. The Trial Input is Unicode (see the setting above)
    OR
  2. You are using filters to convert text to Unicode form (UTF-16LE).

Synchronize Scrolling

When checked, moving the scroll bar of the Trial Input area moves the Trial Output scroll bar as well. This is handy for viewing filter results.

Move Trial Output to Trial Input

Identical to an item in the Tools Menu, this moves the text in the Trial Output area to the Trial Input area.

See also

Drag and Drop
 Filter Lists


Drag and Drop

 Drag and Drop

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Files or folders dropped from Windows Explorer onto TextPipe's window are automatically placed in the files list. Folders have "*.*" automatically appended, so all files in that folder will be processed.

If you drag and drop a TextPipe .fll filter file it will automatically be opened.

See also

File Lists
 Filter Lists


Status Tab

 Status Tab

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up
Errors Tab

The Status Tab of the Processing Files dialog shows you which file is currently being processed, how many files have been processed, how many files have been skipped (due to Input Filter binary file processing options), the progress in this file, and the overall task progress.

Delete all test files/Restore .BAK files

When Test Mode is enabled, clicking this button removes all test files (.DCV).

When Backup Mode is enabled, clicking this button restores all .BAK files.

Copy lines to log file

This button copies result and error lines to the log file if they have not already been copied there based on logging settings.

Cancel

Clicking on the [Cancel] button stops the processing job at the next logical break point (so it may take a short time before it responds).

Ok

When the job finished, the [Cancel] button changes to an [Ok] button. Clicking the [Ok] button closes the window.

Processing file

The name of the file currently being processed

Files processed

The number of files that have been processed so far, including the current file

Binary files skipped

The number of files that have been skipped because they appeared to be binary files.

Time elapsed

The amount of time that has elapsed since the filter began processing. This includes time taken waiting for the user to respond to prompting, for example, querying a search and replace.

See also

Drag and Drop
 Filter Lists


Results Tab

 Results Tab

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

The Results Tab of the Processing Files dialog shows you processing results.

The original file size, the new file size and the change in file size (delta or d) is also shown. You can quickly tell if a task has worked properly by comparing the old and new sizes. If you are expecting all files to become larger (e.g. for email encoding), then you expect the Delta column to contain '<'s. If all files are expected to become smaller (e.g. for removing HTML), then you expect the Delta column to contain '>'s. If you expect all files to stay the same size (e.g. for a case conversion, or ASCII <=> EBCDIC), then you expect the Delta column to contain '='s.

Double clicking the filename will open the file in its associated application. If there is no known association, the file is opened with Notepad. This can be changed using the Preferences dialog. The columns widths can be resized by dragging the column separator in the header.

If the filter was being run in Test Mode (i.e. files with a .dCV output), then an extra button will be displayed after the job has been completed giving you the option to delete all generated files. Once the files have been deleted, the button is dimmed.

If more than 1000 files are processed, file logging to the screen is turned off. All files information so far is copied to textPipe.log, and all information about all remaining files processed is also written to the log file.

See also

Drag and Drop
 Filter Lists


Errors Tab

 Errors Tab

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

The Errors Tab of the Processing Files dialog shows you any error messages that occurred as a result of processing.

The Errors Tab is only visible if errors occurred. If more than 100 errors occur, the Errors Tab title will change to 'Errors (Lots)', and error logging to the screen is turned off. All errors logged so far are copied to textPipe.log, and all remaining errors are also written to the log file.

See also

Drag and Drop
 Filter Lists


Input Filter

 Input Filter

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

The Input filter specifies where TextPipe obtains input from.

Input from file(s)

When this radio button is selected, the files given on the "Files to Process" tab will each be used in turn as a source of input data.

    Binary files

    Generally it isn’t useful to process binary files, because the application they are created by imposes structure on them (unlike text files). If you want all files to be processed (regardless of whether they appear to be binary), choose "Process". If you don’t want any binary files to be processed, choose "Skip". To be given a choice to process a file when TextPipe thinks that it is binary, choose "Prompt before processing".

    To determine if a file is binary, TextPipe applies the following rules:

    1. If four characters in a row have an ASCII code between 128 and 255, TextPipe immediately assumes the file is binary;
    2. If any character read has an ASCII code less than 32 but is not one of the standard formatting codes of 9,10,12,13,26 or 27, TextPipe immediately assumes the file is binary;
    3. If less than 80% of characters in the sample are readable, TextPipe assumes the file is binary. The number of characters in the sample can be configured using the Sample Size field.

    This option can be enabled on the command line by using the /B=P (process binary files) or /B=S (skip binary files) option. The option to prompt cannot be chosen from the command line.

    Sample Size

    If you find that your binary files are not being identified correctly, you can increase the sample size of text that TextPipe examines to determine if a file is binary.

    Confirm before processing each file

    TextPipe will ask the user to confirm processing each of read only file that it encounters.

    This option cannot be enabled from the command line.

    Confirm before processing read only files

    Before processing each file, TextPipe prompts the user to confirm that the file should be processed.

    This option cannot be enabled from the command line.

    Delete input files after processing

    This option deletes the input files after they have been processed. It is useful when the original files are not required after transformation, but should be used with EXTREME CAUTION.

    This option cannot be enabled from the command line.

Input from clipboard

When this radio button is selected, the current contents of the Windows clipboard is used as the source of input data.

Input from Trial Run Input

When this radio button is selected, TextPipe uses the contents of the Trial Run Input Area for its input. This is very handy to avoid having to save a text file to provide the input.

Generate Random Input of Size ...

This filter is used to generate files of a specific size, containing purely random data. The resulting file can be used as a test file for other applications. The data is generated by a pseudo-random number generator, and is different each time the filter is run. The characters generated are in the full ASCII range of 0 to 255 and have a normal distribution. The maximum file size than can be generated is 2147483647 bytes (2 GB).

See also

Output filter


Output Filter

 Output Filter

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter specifies where TextPipe sends output to.

Output to file(s)

Output Mode

This option controls how the original files are preserved.

Normal mode - the original files are overwritten with the processed results. This is the default.

Test mode - processed files are output to a .dCv extension

The processed file is the original file's name with '.dCv' added to it. This is handy to ensure that you don’t destroy your original file. You can quickly delete the test files by using the [Delete Test Files] button in the results dialog before closing it. This option can be enabled on the command line by using the /T+ option.

Backup mode - The original files are renamed by adding a .bak extension

The original file is renamed to a .bak extension, and the processed file is output to the original filename. This is handy to ensure that you don’t destroy your original file, and this option is highly recommended. You can quickly restore the original files by using the [Restore .BAK Files] button in the results dialog before closing it. This option can be enabled on the command line by using the /TB option.

Only output modified files

When this option is selected, only files that are changed by the filter list will be marked as changed with a new date. This makes it easier to find which files have been affected or not.

Keep original file’s date/time

The processed file is given the same date and time as the original file.

This option can be enabled on the command line by using the /N option. Retaining the date and time is disabled by default.

Append to existing file

When checked, this setting will append output to the existing output file rather than first deleting the existing output file.

Change extension to

Changes the extension of the output file to the given value when selected. The extension given can optionally include a leading period '.', one will be added if it does not have one. This option is useful when you are converting from one file type to another e.g. HTML to .txt, .pas to .c etc.

Output folder

Changes the folder that the output files are stored in. This is useful if you want to keep the new files completely separate from the old files, or if you do not have permission to write to the folder where the original files are kept.

A backslash '\' is added to the folder name if it does not have one.

Maintain folder structure

If a new output folder is selected, all files will be output into the specified folder regardless of their original folder. If the Maintain folder structure check box is selected, the original folder structure is maintained beneath the starting directory.

Output to clipboard (join input files together)

When this radio button is selected, all output is joined together and sent to the Windows clipboard.

Single file Output (join input files together)

When this radio button is selected, all output is joined together and sent to the specified file. If you need to join files together and then perform further text processing, such as adding headers, footers, sorting and/or removing duplicates, then use a separate Merge Filter instead of using this option. Set the Output Filter to File(s) Output mode.

See also

Input filter


File Menu

 File Menu

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up
Link to TextPipe filter file (.fll)
TextPipe Sample Filter

New

Opens a blank new filter window. New filters are created using the normal template, normal.fll, found in TextPipe's installation folder.

Open

Opens a saved filter list file.

To open a sample filter:

  1. Click on Open in the File menu
  2. TextPipe displays the filter list files in the TextPipe folder, usually c:\Program Files\TextPipe. Navigate to the TextPipe folder if the folder displayed is not the TextPipe folder
  3. Select a filter list file (.fll file) to open, or navigate into one of the subject area subfolders such as coding, conversions, email, etext, fun, html or sysadmin and select a filter list file to open
  4. Click the Ok button
  5. TextPipe will open and display the selected filter list file.

Open in same window

Opens a filter in the same edit window, without modifying the file list. This can be used if you load a file list first either by drag and drop or from the Window Explorer context menu, and then need to change the filter without modifying the files.

Search

Opens a Windows Explorer search dialog box in TextPipe's installation folder. This can be used to quickly search for filters.

Insert

Inserts a saved filter list at the current cursor position.

Sometimes you need to incorporate the filters that you have saved in a filter list file. To copy them in to the current filter:

  1. Position the cursor on top of the filter where you want the inserted filter list to appear. The inserted filter list will appear ABOVE the currently selected filter
  2. Click Insert in the File menu
  3. TextPipe displays the filter list files in the TextPipe folder, usually c:\Program Files\TextPipe. Navigate to the TextPipe folder if the folder displayed is not the TextPipe folder
  4. Select a filter list file (.fll file) to insert, or navigate into one of the subject area subfolders such as coding, conversions, email, etext, fun, html or sysadmin and select a filter list file to insert
  5. Click the Ok button
  6. TextPipe will insert the new filter list above the currently selected filter.

Save

Saves the current filter list. If no name has been given to it, a Save As is performed.

Save as

Saves the current filter list with a new name.

Close

Closes the current filter window (prompting to save if any changes have been made).

Close all

Closes all window (prompting to save if any changes have been made).

Link to filter (Pro version only)

Incorporates the filters found in an external filter file at run time. This means than any changes that have been made to the linked filter list will be reflected in any filter list that links to it.

To link to a filter list:

  1. Position the cursor on top of the filter where you want the linked filter list to appear. The linked filter list will appear BELOW the currently selected filter
  2. Click Link to filter in the File menu (Pro version only)
  3. TextPipe displays the filter list files in the TextPipe folder, usually C:\Program Files\TextPipe. Navigate to the TextPipe folder if the folder displayed is not the TextPipe folder
  4. Select a filter list file (.fll file) to link to, or navigate into one of the subject area subfolders such as coding, conversions, email, etext, fun, html or sysadmin and select a filter list file to link
  5. Click the Ok button
  6. TextPipe will insert a new link filter BELOW the currently selected filter.

Notes: When you link to a filter file, its Input and Output Filters are ignored, and all its Files to Process are ignored. Instead, the Input and Output filters are overridden by the Input and Output Filters of the filter list linking to it.

If the linked filter list performs special output functions in the Output Filter, such as merging the result to a new file, then when it is linked this function will no longer work (because the Output Filter is ignored). You cannot use a linked filter to re-process the original files in their unmodified state - whatever the linked filter does (including merging files together) is passed on to the next filter. The best way to achieve this is to run two separate filters consecutively from a batch file.

Export

Export filter to clipboard (Pro version only)

Uses the same output format as printing the filter list, but sends it to the clipboard so that it can be pasted into other programs. This feature is a very handy way of maintaining text-based revision control of your filters. Using a text-based diff program you can compare two filters easily after exporting each to the clipboard.

Export command line (Pro version only)

Generates command line parameters corresponding to the current filter list, and allows them to be saved to a batch file (.BAT). Note that not all filters are supported via the command line. This can be handy when editing complex filters externally to TextPipe. Note that the hierarchical structure of the filter list is NOT maintained.

Export to VBScript (Pro version only)

Generates VBScript code corresponding to the entire current filter list, and allows it to be saved to a VBScript file (.VBS). ALL filters are supported, and the hierarchical structure of the filter list IS maintained. This can be very handy when editing complex filters externally to TextPipe, and for creating code to inserting into Web pages, ASP scripts or other scripting application.

Export to JScript (Pro version only)

Generates JScript code corresponding to the entire current filter list, and allows it to be saved to a JScript file (.JS). ALL filters are supported, and the hierarchical structure of the filter list IS maintained. This can be very handy when editing complex filters externally to TextPipe, and for creating code to inserting into Web pages, ASP scripts or other scripting application.

Export to web page (Pro version only)

Generates a sample web page with both VBScript and JScript code corresponding to the entire current filter list, and allows it to be saved to a web page (.HTM). Sample edit boxes are also provided for testing the filter on text. This can be very handy as a template to create your own TextPipe web-based front-end - making it easier for non-technical users.

Print

Prints the current filter list. The output shows the names of the filters and the option settings, and the file list.

Printer setup

Allows the page orientation etc to be configured for the printer.

Recent files

Recent files is a submenu that shows a list of previously opened filter files, if any. A previous filter file can be opened quickly by just clicking on it in the menu. The maximum number of entries shown can be controlled through the preferences dialog.

Exit

Leaves TextPipe. You will be prompted to save the current filter list if you have modified it. Clicking [Yes] saves the filter and exits, clicking [No] does not save the filter and still exits, and clicking on [Cancel] does not exit.


Edit Menu

 Edit Menu

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up
Preferences Dialog

Undo changes to this line

Restores the previous values for filter settings displayed in the right-hand window.

Cut

Cuts selected text to the Windows clipboard. If the current control is the filter tree, this cuts the selected filter and its sub filters to the clipboard.

Copy

Copies selected text to the Windows clipboard. If the current control is the filter tree, this copies the selected filter and its sub filters to the clipboard. To Copy an entire filter tree to the clipboard, right click the top-most node (logging options) and choose Copy.

Paste

Pastes text from the Windows clipboard to the current cursor position. TextPipe escapes special characters e.g. for a single line edit control, carriage return/line feeds are escaped to \r\n. If the current control is the filter tree, and a filter has been Cut or Copied to the clipboard, this pastes the filters into the filter tree.

Paste special

Pastes text from the Windows clipboard without any intervention from TextPipe (normally TextPipe escapes special characters e.g. for a single line edit control, carriage return/line feeds are escaped to \r\n).

Delete

Deletes the currently selected text.

Duplicate current filter (x2)

Copies the filter underneath the cursor (and all of its sub filters) and inserts it below the current cursor position. The [x2] button can also be clicked to perform the same action.

Clear clipboard

Clear the current clipboard contents.

Select All

Selects all the text in the current control. Has no effect if the current control is the filter tree. To Copy an entire filter tree to the clipboard, right click the top-most node (logging options) and choose Copy.

Toggle filter enable/disable

Toggles the filter under the cursor between an enabled and disabled state. When disabled, a filter takes no part in filter processing. Disabled filters are shown with a red cross over their icon. This menu item is only available when the Filters to Apply tab is displayed.

Filters can also be toggled by double clicking on them in the filter list.

Preferences

Displays the preferences dialog


Conversion Filters

 Conversion Filters

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up
Convert End of Lines (EOL)
IBM Drawing Characters
Tabs to Spaces
Spaces to Tabs
UPPERCASE
lowercase
Title Case
Sentence case
tOGGLE cASE
raNDoM cASe
ASCII to EBCDIC
EBCDIC to ASCII
Expand Packed Decimal
Expand Zoned Decimal
ASCII/OEM to ANSI
ANSI to ASCII/OEM
Unicode conversion
ANSI to Unicode (UTF-16LE)
Unicode (UTF-16LE) to ANSI
Make Big Endian
Make Little Endian
Swap UTF-16 word order
Swap UTF-32 word order
CSV to Tab-delimited
CSV to XML
Tab-delimited to CSV
Tab-delimited to XML
Text to Word List
Word documents to text files
Hex dump
Decimal dump
Character maths
Compress to Packed Decimal
Compress to Zoned Decimal

Found under the "Convert" menu, these filters generally do not delete or add information, but change it into a different form.

Convert End of Line Characters
Convert IBM Drawing Characters

Tabs and spaces

Tabs to Spaces
Spaces to Tabs

Character Case

UPPERCASE
lowercase
Title Case
Sentence case.
tOGGLE cASE
raNDoM cASe

Mainframe

ASCII to EBCDIC (Pro version only)
EBCDIC to ASCII (Pro version only)
Expand packed decimal (Pro version only)
Expand packed numeric (use the Map Expand packed numeric.map)
Expand zoned decimal (Pro version only)
Compress packed decimal (Pro version only)
Compress zoned decimal (Pro version only)
Records (same as Convert end of line characters)

Character set

ASCII/OEM to ANSI (Pro version only)
ANSI to ASCII/OEM (Pro version only)

Unicode - now found under the Unicode Menu

CSV

CSV to Tab-delimited (Pro version only)
CSV to XML (Pro version only)

Tab-delimited

Tab-delimited to CSV (Pro version only)
Tab-delimited to XML (Pro version only)

Text to Word List (Pro version only)
Word documents to text files (Pro version only)

Hex dump (Pro version only)
Decimal dump (Pro version only)
Character maths (Pro version only)


Convert End of Lines (EOL)

 Convert End of Lines (EOL)

Submit feedback on this topic

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

The Operating Systems used by various computers do not share a standard symbol that represents the end of each line in text files. Unix and the Amiga both use ASCII code 10, Macintoshes use ASCII code 13, and PCs use ASCII code 13 followed by 10.

This difference introduces an extra complication into transferring files between computers – the end of line characters must be converted to match the target system. Mainframes do not use end of line characters, but instead assume a fixed length for each line.

Input line feeds

Select the line feed type of the input files.

Auto

If you are unsure of the source system, select the "Auto" option. The "Auto" option can reliably detect and correct text files containing a variety of line feeds.

Fixed length

If you are converting a mainframe file that contains fixed length records, select "Fixed length" and enter the record length. The maximum record length is 2147483647.

Note: If you are converting 132 column mainframe reports, you should set the fixed length to 133, because each line has a prefix character.

Mainframe CMS Format

This setting converts variable length EBCDIC data to ASCII delimited lines, each terminated by the selected Output line feed character.

Remove bad EOLs

If you suspect that the input file contains different end of line symbols than it should (eg a Unix text file containing ASCII code 13s), then check "Remove Bad EOLs". Note that this option can be dangerous – because if the file has no line feeds of the type that you select, all line feeds will be removed. Always use a test run first to make sure.

Output line feeds

Select the line feed type to output. If you want all line feeds to be removed, select "None". If you want to specify your own character or group of characters, select the radio button beside the edit field and enter the text you wish to be used. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy and entering special characters.


IBM Drawing Characters

 IBM Drawing Characters

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

IBM drawing characters in the upper ASCII range (128-255) are commonly used to draw lines and boxes, single and double line borders, shaded characters etc. Many devices (eg printers, non-IBM computers etc) do not support the display of these characters.

This filter converts them to standard ASCII characters (+, - and |) that all computers can display.


Tabs to Spaces

 Tabs to Spaces

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

number_filter.gif (2952 bytes)

Expands tabs out into the equivalent number of spaces, usually 8. This filter is used to make files easier to process with programs that support differing tab sizes.

Some text editors expect a specific tab size, and save files with the tabs embedded. To import these files successfully into text editors that use a different tab size, you need to use this filter to first convert the tabs to spaces.

This filter may increase the size of a file.

See also:

  • Spaces to Tabs


Spaces to Tabs

 Spaces to Tabs

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Reduces spaces into the appropriate number of tabs. There are usually 8 spaces per tab. This filter is used to make files smaller.

See also:

  • Tabs to Spaces


UPPERCASE

 UPPERCASE

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Forces all text to UPPERCASE. This routine is ANSI-aware (handles foreign character sets).

See also:

lowercase
Title Case
Sentence case
tOGGLE cASE
raNDoM cASe


lowercase

 lowercase

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Forces all text to lowercase. This routine is ANSI-aware (handles foreign character sets).

See also:

UPPERCASE
Title Case
Sentence case
tOGGLE cASE
raNDoM cASe


Title Case

 Title Case

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Converts all text to Title Case ie the first letter of every word is capitalised, and all other letters are forced to lower case. This routine is ANSI-aware (handles foreign character sets).

See also:

UPPERCASE
lowercase
Sentence case
tOGGLE cASE
raNDoM cASe


Sentence case

 Sentence case

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Converts all text to Sentence case ie the first word in every sentence is capitalised, all other letters are left as is. Sentences are deemed to start after full stops, exclamation marks, colons, question marks, quotes, brackets and angle brackets (.!:?'"<(). This routine is ANSI-aware (handles foreign character sets).

See also:

UPPERCASE
lowercase
Title Case
tOGGLE cASE
raNDoM cASe


tOGGLE cASE

 tOGGLE cASE

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

tOGGLES tHE cASE of all text ie all UPPERCASE characters are converted to lowercase and vice-versa. This routine is ANSI-aware (handles foreign character sets).

See also:

UPPERCASE
lowercase
Title Case
Sentence case
raNDoM cASe


raNDoM cASe

 raNDoM cASe

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter randomly changes the case of characters. This routine is ANSI-aware (handles foreign character sets).

Running this filter again will generate different results e.g.

  1. ranDoMIze cASe
  2. RanDOmIZE case
  3. randOMIZE casE

See also:

UPPERCASE
lowercase
Title Case
Sentence case
tOGGLE cASE


ASCII to EBCDIC

 ASCII to EBCDIC

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

Converts from ASCII to EBCDIC. EBCDIC is the character collating sequence (order) commonly used on mainframes. Some characters cannot be converted because they exist in one character set but not the other.

EBCDIC to ASCII and ASCII to EBCDIC conversion is usually done by the FTP program that performs the file transfer or download, but sometimes you may receive a file in the wrong format and there is no easy way to convert it. When the EBCDIC data file is moved down, ensure that it has correct blocking (mainframe talk for correct placement of CR\LF at end of every line). This is normally done when the file is moved down. Alternatively, use the Convert End Of Lines filter to put in line feeds manually.

If the ASCII to EBCDIC conversion doesn't suit you perfectly, use the Base Map On Filter List command to customize the translation.

See also our mainframe conversion guide online in the documents area.

See also:

EBCDIC to ASCII
Expand Zoned Decimal
Expand Packed Decimal
Tutorials and User Guides (online)


EBCDIC to ASCII

 EBCDIC to ASCII

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

Converts from EBCDIC to ASCII. EBCDIC is the character collating sequence (order) commonly used on mainframes. Some characters cannot be converted because they exist in one character set but not the other.

If the EBCDIC to ASCII conversion doesn't suit you perfectly, use the Base Map On Filter List command to customize the translation.

See also our mainframe conversion guide online in the documents area.

See also:

ASCII to EBCDIC
Expand Zoned Decimal
Expand Packed Decimal
Tutorials and User Guides (online)


Expand Packed Decimal

 Expand Packed Decimal

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter expands an EBCDIC packed decimal field (also known as Comp-3) to a raw EBCDIC number with a sign. Typically this filter is then followed by a Convert EBCDIC to ASCII filter - after all other fields have been expanded as well.

There are several notes to keep in mind when using this filter:

  1. You MUST use this filter inside a Restrict to Byte Range filter. The field WIDTH is then set by the containing filter.
  2. Expanding a field will increase your output record length, so ensure you allow for this. A good strategy to avoid problems is to first expand the rightmost field, then work your work back to the leftmost field. This prevents the field column positions from changing and makes the file easier to work with.

(*) Performance Note

If your numbers are always positive (i.e. no negatives) and there are no implied decimals, you will get far better performance using the map Expand EBCDIC Packed Decimal.

Handling Massive Data Extracts

If you are exporting massive amounts of data from one system to import into another, you can use the following strategies to minimize file size and get the file in the right format:

  1. To skip a header record, add a Remove start or end of file filter.
  2. Use a Restrict to blocks of length filter, with the size set to your record size. Inside this filter, expand all packed and zoned fields from right to left, convert EBCDIC to ASCII, and then add a CR/LF.
  3. Insert commas before all ALPHA fields (EBCDIC \x6B). Note - field positions may have moved.
  4. To remove leading zeroes and add commas to numeric fields:
    Search for perl pattern | Replace with
    -0*?([0-9]) | ,-$1
    \+0*?([0-9]) | ,$1
  5. If you end up with a comma at the start of each line, use a remove columns filter.

All these filters can be found in the sample filter mainframe\generic mainframe file.fll

See also our mainframe conversion guide online in the documents area.

See also:

EBCDIC to ASCII
ASCII to EBCDIC
Expand Zoned Decimal
Tutorials and User Guides (online)


Expand Zoned Decimal

 Expand Zoned Decimal

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter expands an EBCDIC zoned decimal field to a raw EBCDIC number with a sign. Typically this filter is then followed by a Convert EBCDIC to ASCII filter - after all other fields have been expanded as well.

There are several notes to keep in mind when using this filter:

  1. You MUST use this filter inside a Restrict to Byte Range filter. The field WIDTH is then set by the containing filter.
  2. Expanding a field will increase your output record length, so ensure you allow for this. A good strategy to avoid problems is to first expand the rightmost field, then work your work back to the leftmost field. This prevents the field column positions from changing and makes the file easier to work with.

(*) Performance Note

If your numbers are always positive (i.e. no negatives) and there are no implied decimals, you will get far better performance using the map Expand EBCDIC Zoned Decimal.

Handling Massive Data Extracts

If you are exporting massive amounts of data from one system to import into another, you can use the following strategies to minimize file size and get the file in the right format:

  1. To skip a header record, add a Remove start or end of file filter.
  2. Use a Restrict to blocks of length filter, with the size set to your record size. Inside this filter, expand all packed and zoned fields from right to left, convert EBCDIC to ASCII, and then add a CR/LF.
  3. Insert commas before all ALPHA fields (EBCDIC \x6B). Note - field positions may have moved.
  4. To remove leading zeroes and add commas to numeric fields:
    Search for perl pattern | Replace with
    -0*?([0-9]) | ,-$1
    \+0*?([0-9]) | ,$1
  5. If you end up with a comma at the start of each line, use a remove columns filter.

All these filters can be found in the sample filter mainframe\generic mainframe file.fll

See also our mainframe conversion guide online in the documents area.

See also:

EBCDIC to ASCII
ASCII to EBCDIC
Expand Packed Decimal
Tutorials and User Guides (online)


Compress to Packed Decimal

 Compress to Packed Decimal

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter compresses EBCDIC numeric data (optional leading sign, numbers and periods) to an EBCDIC packed decimal field (also known as Comp-3).

There are several notes to keep in mind when using this filter:

  1. You MUST use this filter inside a Restrict to Byte Range filter. The field WIDTH is then set by the containing filter.
  2. Compressing a field will decrease your output record length, so ensure you allow for this. A good strategy to avoid problems is to first compress the rightmost field, then work your work back to the leftmost field. This prevents the field column positions from changing and makes the file easier to work with.

See also our mainframe conversion guide online in the documents area.

See also:

Compress to Zoned Decimal
EBCDIC to ASCII
ASCII to EBCDIC
Expand Zoned Decimal
Tutorials and User Guides (online)

Contact Us Support Community Tutorials and User Guides (online)
Copyright © 1999-2005 Crystal Software Australia . All rights reserved.


Compress to Zoned Decimal

 Compress to Zoned Decimal

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter compresses EBCDIC numeric data (optional leading sign, numbers and periods) to an EBCDIC zoned decimal field.

There are several notes to keep in mind when using this filter:

  1. You MUST use this filter inside a Restrict to Byte Range filter. The field WIDTH is then set by the containing filter.
  2. Compressing a field will decrease your output record length, so ensure you allow for this. A good strategy to avoid problems is to first expand the rightmost field, then work your work back to the leftmost field. This prevents the field column positions from changing and makes the file easier to work with.

See also our mainframe conversion guide online in the documents area.

See also:

Compress to Packed Decimal
EBCDIC to ASCII
ASCII to EBCDIC
Expand Packed Decimal
Tutorials and User Guides (online)

Contact Us Support Community Tutorials and User Guides (online)
Copyright © 1999-2005 Crystal Software Australia . All rights reserved.


ASCII/OEM to ANSI

 ASCII/OEM to ANSI

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

Converts from ASCII/OEM to ANSI. ANSI is the character set used by Windows, and it includes all accentuated Roman characters used by non-English languages like French, German and Spanish. ASCII/OEM is an extension of the original IBM character set where various non-essential characters are replaced by language-specific accentuated characters. Different ASCII/OEM character sets are not compatible. They must be converted to ANSI and then back to the correct ASCII/OEM character set to be readable.

This filter can be useful if you want to send a text file to someone using a language other than your own.

Also known as a DOS to ANSI filter.

See also:

ANSI to ASCII/OEM


ANSI to ASCII/OEM

 ANSI to ASCII/OEM

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

Converts from ANSI to ASCII/OEM. ANSI is the character set used by Windows, and it includes all accentuated Roman characters used by non-English languages like French, German and Spanish. ASCII/OEM is an extension of the original IBM character set where various non-essential characters are replaced by language-specific accentuated characters. Different ASCII/OEM character sets are not compatible. They must be converted to ANSI and then back to the correct ASCII/OEM character set to be readable.

This filter can be useful if you want to send a text file to someone using a language other than your own.

Also known as an ANSI to DOS filter.

See also:

ASCII/OEM to ANSI


Unicode Conversion

 Unicode conversion

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro only). This filter converts a variety (145) of text encodings between a variety of formats, including to and from Unicode. Unicode itself can be stored in a variety of formats, including UTF-* and UCS-*, with the most common being UTF-16 and UTF-8.

If a character cannot be converted to the destination format, a space is output instead. This maintains the column spacing for reports.

Conversions offered

ARMSCII-8
ASCII
ANSI
BIG5
BIG5HKSCS
C99
CES-BIG5
CES-GBK
CNS-11643-1
CNS-11643-15
CNS-11643-2
CNS-11643-3
CNS-11643-4
CNS-11643-5
CNS-11643-6
CNS-11643-7
CNS-11643-INV
CP1046
CP1124
CP1125
CP1129
CP1133
CP1161
CP1162
CP1163
CP1250
CP1251
CP1252
CP1253
CP1254
CP1255
CP1256
CP1257
CP1258
CP437
CP737
CP775 		CP850
CP852
CP853
CP855
CP856
CP857
CP858
CP860
CP861
CP862
CP863
CP864
CP865
CP866
CP869
CP874
CP922
CP932
CP949
CP950
DEC-Hanyu
DEC-Kanji
EUC-CN
EUC-jisx0213
EUC-JP
EUC-KR
EUC-TW
GB18030
GB2312
GBK
GEORGIAN-ACADEMY
GEORGIAN-PS
HP-ROMAN8
HZ
ISO-2022-CN
ISO-2022-CN-EXT 		ISO-2022-JP
ISO-2022-JP1
ISO-2022-JP2
ISO-2022-JP3
ISO-2022-KR
ISO646-CN
ISO-646-JP
ISO-8859-1
ISO-8859-10
ISO-8859-13
ISO-8859-14
ISO-8859-15
ISO-8859-16
ISO-8859-2
ISO-8859-3
ISO-8859-4
ISO-8859-5
ISO-8859-6
ISO-8859-7
ISO-8859-8
ISO-8859-9
ISO-IR-165
JAVA
JIS_X0201
JIS_X0208
JIS_X0212
JOHAB
Johab-Hangul
KOI8-R
KOI8-RU
KOI8-T
KOI8-U
KSC-5601
MacArabic
MacCentralEurope
MacCroatian 		MacCyrillic
MacGreek
MacHebrew
MacIceland
MacRoman
MacRomania
MacThai
MacTurkish
MacUkraine
MULELAO-1
NEXTSTEP
RISCOS1
SHIFT-JISX0213
SJIS
TCVN
TDS565
TIS-620
UCS-2
UCS-2BE
UCS-2-Internal
UCS-2LE
UCS-2-Swapped
UCS-4
UCS-4BE
UCS-4-Internal
UCS-4LE
UCS-4-Swapped
UTF-16
UTF-16BE
UTF-16LE
UTF-32
UTF-32BE
UTF-32LE
UTF-7
UTF-8
VISCII

Note:

  • UCS-4 is UTF-32 with support for code points beyond U+10FFFF (which are supposed to be unassignable forever).
  • UCS-2 is UTF-16 with surrogate support removed (so code points beyond U+FFFF cannot be represented).

Equivalent encodings

The following list gives equivalent encodings (different names for the same encoding), one per line.

US-ASCII,ASCII,ISO646-US,ISO_646.IRV:1991,ISO-IR-6,ANSI_X3.4-1968,ANSI_X3.4-1986,CP367,IBM367,US,csASCII
UCS-2,ISO-10646-UCS-2,csUnicode
UCS-2BE,UNICODEBIG,UNICODE-1-1,csUnicode11
UCS-2LE,UNICODELITTLE
UCS-4,ISO-10646-UCS-4,csUCS4
UTF-7,UNICODE-1-1-UTF-7,csUnicode11UTF7
ISO-8859-1,ISO_8859-1,ISO_8859-1:1987,ISO-IR-100,CP819,IBM819,LATIN1,L1,csISOLatin1,ISO8859-1
ISO-8859-2,ISO_8859-2,ISO_8859-2:1987,ISO-IR-101,LATIN2,L2,csISOLatin2,ISO8859-2
ISO-8859-3,ISO_8859-3,ISO_8859-3:1988,ISO-IR-109,LATIN3,L3,csISOLatin3,ISO8859-3
ISO-8859-4,ISO_8859-4,ISO_8859-4:1988,ISO-IR-110,LATIN4,L4,csISOLatin4,ISO8859-4
ISO-8859-5,ISO_8859-5,ISO_8859-5:1988,ISO-IR-144,CYRILLIC,csISOLatinCyrillic,ISO8859-5
ISO-8859-6,ISO_8859-6,ISO_8859-6:1987,ISO-IR-127,ECMA-114,ASMO-708,ARABIC,csISOLatinArabic,ISO8859-6
ISO-8859-7,ISO_8859-7,ISO_8859-7:1987,ISO-IR-126,ECMA-118,ELOT_928,GREEK8,GREEK,csISOLatinGreek,ISO8859-7
ISO-8859-8,ISO_8859-8,ISO_8859-8:1988,ISO-IR-138,HEBREW,csISOLatinHebrew,ISO8859-8
ISO-8859-9,ISO_8859-9,ISO_8859-9:1989,ISO-IR-148,LATIN5,L5,csISOLatin5,ISO8859-9
ISO-8859-10,ISO_8859-10,ISO_8859-10:1992,ISO-IR-157,LATIN6,L6,csISOLatin6,ISO8859-10
ISO-8859-13,ISO_8859-13,ISO-IR-179,LATIN7,L7,ISO8859-13
ISO-8859-14,ISO_8859-14,ISO_8859-14:1998,ISO-IR-199,LATIN8,L8,ISO-CELTIC,ISO8859-14
ISO-8859-15,ISO_8859-15,ISO_8859-15:1998,ISO-IR-203,ISO8859-15
ISO-8859-16,ISO_8859-16,ISO_8859-16:2000,ISO-IR-226,ISO8859-16
KOI8-R,csKOI8R
CP1250,WINDOWS-1250,MS-EE
CP1251,WINDOWS-1251,MS-CYRL
CP1252,WINDOWS-1252,MS-ANSI
CP1253,WINDOWS-1253,MS-GREEK
CP1254,WINDOWS-1254,MS-TURK
CP1255,WINDOWS-1255,MS-HEBR
CP1256,WINDOWS-1256,MS-ARAB
CP1257,WINDOWS-1257,WINBALTRIM
CP1258,WINDOWS-1258
CP850,IBM850,850,csPC850Multilingual
CP862,IBM862,862,csPC862LatinHebrew
CP866,IBM866,866,csIBM866
MACINTOSH,MAC,csMacintosh
HP-ROMAN8,ROMAN8,R8,csHPRoman8
CP1133,IBM-CP1133
TIS-620,TIS620,TIS620-0,TIS620.2529-1,TIS620.2533-0,TIS620.2533-1,ISO-IR-166
CP874,WINDOWS-874
VISCII,VISCII1.1-1,csVISCII
TCVN,TCVN-5712,TCVN5712-1,TCVN5712-1:1993
JIS_C6220-1969-RO,ISO646-JP,ISO-IR-14,JP,csISO14JISC6220ro
JIS_X0201,JISX0201-1976,X0201,csHalfWidthKatakana
JIS_X0208,JIS_X0208-1983,JIS_X0208-1990,JIS0208,X0208,ISO-IR-87,JIS_C6226-1983,csISO87JISX0208
JIS_X0212,JIS_X0212.1990-0,JIS_X0212-1990,X0212,ISO-IR-159,csISO159JISX02121990
GB_1988-80,ISO646-CN,ISO-IR-57,CN,csISO57GB1988
GB_2312-80,ISO-IR-58,csISO58GB231280,CHINESE
ISO-IR-165,CN-GB-ISOIR165
KSC_5601,KS_C_5601-1987,KS_C_5601-1989,ISO-IR-149,csKSC56011987,KOREAN
EUC-JP,EUCJP,Extended_UNIX_Code_Packed_Format_for_Japanese,csEUCPkdFmtJapanese
SHIFT_JIS,SHIFT-JIS,SJIS,MS_KANJI,csShiftJIS
ISO-2022-JP,csISO2022JP
ISO-2022-JP-2,csISO2022JP2
EUC-CN,EUCCN,GB2312,CN-GB,csGB2312
GBK,CP936
ISO-2022-CN,csISO2022CN
HZ,HZ-GB-2312
EUC-TW,EUCTW,csEUCTW
BIG5,BIG-5,BIG-FIVE,BIGFIVE,CN-BIG5,csBig5
BIG5-HKSCS,BIG5HKSCS
EUC-KR,EUCKR,csEUCKR
CP949,UHC
JOHAB,CP1361
ISO-2022-KR,csISO2022KR

See also

Swap UTF-16 word order
Swap UTF-32 word order
Make Big Endian
Make Little Endian
Unicode conversion

Conversions are provided by the libiconv library.


ANSI to Unicode (UTF16LE)

 ANSI to Unicode (UTF-16LE)

Submit feedback on this topic

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

Converts from ANSI to Unicode. Converts single byte ANSI characters to double byte Unicode characters. This filter can be useful if you want to send a text file to someone using a language other than your own.

This function was originally included in older versions of TextPipe. To use this feature in conjunction with the new Unicode conversion functions, you must first convert from UTF-16LE to Unicode.

On our test system, US ASCII is transformed to UTF-16LE:

00000000 20 00 09 00 20 00 33 00 32 00 0D 00 0A 00 21 00 ... .3.2.....!.
00000010 09 00 20 00 33 00 33 00 0D 00 0A 00 22 00 09 00 .. .3.3....."...
00000020 20 00 33 00 34 00 0D 00 0A 00 23 00 09 00 20 00 .3.4.....#... .
00000030 33 00 35 00 0D 00 0A 00 24 00 09 00 20 00 33 00 3.5.....$... .3.

This filter is often followed by an Add Header filter, to add a Unicode byte order mark (BOM), \xFF\xFE.

See also

Unicode to ANSI
Unicode conversion


Unicode (UTF16LE) to ANSI

 Unicode (UTF-16LE) to ANSI

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

Converts from Unicode to ANSI. Converts double byte Unicode characters to single byte ANSI characters. This filter can be useful if you want to send a text file to someone using a language other than your own.

This function was originally included in older versions of TextPipe. To use this feature in conjunction with the new Unicode conversion functions, you must first convert from UTF-16LE to Unicode.

This filter is often followed by a Remove start or end of file filter, to either remove the first two bytes of Unicode (before the conversion) or the first byte of ANSI (after the conversion), to remove the leading Unicode byte order mark (BOM).

See also

ANSI to Unicode


Swap UTF-16 word order

 Swap UTF-16 word order

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro only). This filter swaps pairs of bytes e.g.

Byte number | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8
Input File | FF | FE | 00 | 20 | 00 | 31 | 00 | 32
Output File | FE | FF | 20 | 00 | 31 | 00 | 32 | 00

This is commonly used to transform big-endian or little-endian Unicode files so that other programs can use them.

See also

Swap UTF-16 word order
Swap UTF-32 word order
Make Big Endian
Make Little Endian
Unicode conversion


Swap UTF-32 word order

 Swap UTF-32 word order

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro only). This filter swaps groups of 2-byte words e.g.

Byte number | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8
Input File | FF | FE | 00 | 00 | 00 | 31 | 00 | 00
Output File | 00 | 00 | FE | FF | 00 | 00 | 31 | 00

This is commonly used to transform big-endian or little-endian Unicode files so that other programs can use them.

See also

Swap UTF-16 word order
Swap UTF-32 word order
Make Big Endian
Make Little Endian
Unicode conversion


Make Big Endian

 Make Big Endian

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro only). Makes a Little Endian Unicode file into a Big Endian Unicode file e.g.

Input file | Output file
00 00 FE FF 00 00 00 4D | Unchanged
FE FF 4E 8C | Unchanged
FF FE 00 00 4D 00 00 00 | 00 00 FE FF 00 00 00 4D
FF FE 8C 4E | FE FF 4E 8C

Note - the file MUST start with a Byte Order Mark for it to be correctly identified.

See also

Swap UTF-16 word order
Swap UTF-32 word order
Make Big Endian
Make Little Endian
Unicode conversion


Make Little Endian

 Make Little Endian

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro only). Makes a Big Endian Unicode file into a Little Endian Unicode file e.g.

Input file | Output file
00 00 FE FF 00 00 00 4D | FF FF 00 00 4D 00 00 00
FE FF 4E 8C | FF FE 8C 4E
FF FE 00 00 4D 00 00 00 | Unchanged
FF FE 8C 4E | Unchanged

Note - the file MUST start with a Byte Order Mark for it to be correctly identified.

See also

Swap UTF-16 word order
Swap UTF-32 word order
Make Big Endian
Make Little Endian
Unicode conversion


CSV to Tab-delimited

 CSV to Tab-delimited

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

Converts CSV data (quoted or unquoted) to tab-delimited form. It's preferable to use a file with column headers, because then TextPipe can easily determine if the fields have embedded CR/LFs in them. If the data is properly quoted then TextPipe will determine this automatically.

e.g. Input CSV file:

1,,3,,5,,7,,9,,11,,13,
"1","2","3","4","5","6","7","8","9","10","11","12","13","14"
1,2,3,4,5,6,7,8,9,10,11,12,13,14
1,2,3,4,5,6,7,8,9,10,11,12,13,
 "1","2","3","4","53424""34242","""3243246","7","8","9","10","11","12","13","14"

Output Tab-delimited file (Tabs are not shown correctly)

1 3 5 7 9 11 13
1 2 3 4 5 6 7 8 9 10 11 12 13 14
1 2 3 4 5 6 7 8 9 10 11 12 13 14
1 2 3 4 5 6 7 8 9 10 11 12 13
1 2 3 4 53424"34242 "3243246 7 8 9 10 11 12 13 14


CSV to XML

 CSV to XML

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

Converts CSV data (quoted or unquoted) to XML form. It's preferable to use a file with column headers, because then TextPipe can easily determine if the fields have embedded CR/LFs in them. If the data is properly quoted then TextPipe will determine this automatically.

e.g. Input CSV file:

1,,3,,5,,7,,9,,11,,13,
"1","2","3","4","5","6","7","8","9","10","11","12","13","14"
1,2,3,4,5,6,7,8,9,10,11,12,13,14
1,2,3,4,5,6,7,8,9,10,11,12,13,
 "1","2","3","4","53424""34242","""3243246","7","8","9","10","11","12","13","14"

Output XML format

<line>
<field1>1</field1>
<field2></field2>
<field3>3</field3>
<field4></field4>
<field5>5</field5>
<field6></field6>
<field7>7</field7>
<field8></field8>
<field9>9</field9>
<field10></field10>
<field11>11</field11>
<field12></field12>
<field13>13</field13>
<field14></field14>
</line>
<line>
<field1>1</field1>
<field2>2</field2>
<field3>3</field3>
<field4>4</field4>
<field5>5</field5>
<field6>6</field6>
<field7>7</field7>
<field8>8</field8>
<field9>9</field9>
<field10>10</field10>
<field11>11</field11>
<field12>12</field12>
<field13>13</field13>
<field14>14</field14>
</line>
<line>
<field1>1</field1>
<field2>2</field2>
<field3>3</field3>
<field4>4</field4>
<field5>5</field5>
<field6>6</field6>
<field7>7</field7>
<field8>8</field8>
<field9>9</field9>
<field10>10</field10>
<field11>11</field11>
<field12>12</field12>
<field13>13</field13>
<field14>14</field14>
</line>
<line>
<field1>1</field1>
<field2>2</field2>
<field3>3</field3>
<field4>4</field4>
<field5>5</field5>
<field6>6</field6>
<field7>7</field7>
<field8>8</field8>
<field9>9</field9>
<field10>10</field10>
<field11>11</field11>
<field12>12</field12>
<field13>13</field13>
<field14></field14>
</line>
<line>
<field1>1</field1>
<field2>2</field2>
<field3>3</field3>
<field4>4</field4>
<field5>53424"34242</field5>
<field6>"3243246</field6>
<field7>7</field7>
<field8>8</field8>
<field9>9</field9>
<field10>10</field10>
<field11>11</field11>
<field12>12</field12>
<field13>13</field13>
<field14>14</field14>
</line>


Tab-delimited to CSV

 Tab-delimited to CSV

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

Converts Tab-delimited data to CSV data. It's preferable to use a file with column headers, because then TextPipe can easily determine if the fields have embedded CR/LFs in them. TextPipe cannot determine this without column headers.

Input Tab-delimited file (Tabs are not shown correctly)

1 3 5 7 9 11 13
1 2 3 4 5 6 7 8 9 10 11 12 13 14
1 2 3 4 5 6 7 8 9 10 11 12 13 14
1 2 3 4 5 6 7 8 9 10 11 12 13
1 2 3 4 53424"34242 "3243246 7 8 9 10 11 12 13 14

Output CSV format

"1","","3","","5","","7","","9","","11","","13",
"1","2","3","4","5","6","7","8","9","10","11","12","13","14"
"1","2","3","4","5","6","7","8","9","10","11","12","13","14"
"1","2","3","4","5","6","7","8","9","10","11","12","13",
"1","2","3","4","53424""34242","""3243246","7","8","9","10","11","12","13","14"


Tab-delimited to XML

 Tab-delimited to XML

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

Converts Tab-delimited data to XML data. It's preferable to use a file with column headers, because then TextPipe can easily determine if the fields have embedded CR/LFs in them. TextPipe cannot determine this without column headers.

Input Tab-delimited file (Tabs are not shown correctly)

1 3 5 7 9 11 13
1 2 3 4 5 6 7 8 9 10 11 12 13 14
1 2 3 4 5 6 7 8 9 10 11 12 13 14
1 2 3 4 5 6 7 8 9 10 11 12 13
1 2 3 4 53424"34242 "3243246 7 8 9 10 11 12 13 14

Output XML format

<line>
<field1>1</field1>
<field2></field2>
<field3>3</field3>
<field4></field4>
<field5>5</field5>
<field6></field6>
<field7>7</field7>
<field8></field8>
<field9>9</field9>
<field10></field10>
<field11>11</field11>
<field12></field12>
<field13>13</field13>
<field14></field14>
</line>
<line>
<field1>1</field1>
<field2>2</field2>
<field3>3</field3>
<field4>4</field4>
<field5>5</field5>
<field6>6</field6>
<field7>7</field7>
<field8>8</field8>
<field9>9</field9>
<field10>10</field10>
<field11>11</field11>
<field12>12</field12>
<field13>13</field13>
<field14>14</field14>
</line>
<line>
<field1>1</field1>
<field2>2</field2>
<field3>3</field3>
<field4>4</field4>
<field5>5</field5>
<field6>6</field6>
<field7>7</field7>
<field8>8</field8>
<field9>9</field9>
<field10>10</field10>
<field11>11</field11>
<field12>12</field12>
<field13>13</field13>
<field14>14</field14>
</line>
<line>
<field1>1</field1>
<field2>2</field2>
<field3>3</field3>
<field4>4</field4>
<field5>5</field5>
<field6>6</field6>
<field7>7</field7>
<field8>8</field8>
<field9>9</field9>
<field10>10</field10>
<field11>11</field11>
<field12>12</field12>
<field13>13</field13>
<field14></field14>
</line>
<line>
<field1>1</field1>
<field2>2</field2>
<field3>3</field3>
<field4>4</field4>
<field5>53424"34242</field5>
<field6>"3243246</field6>
<field7>7</field7>
<field8>8</field8>
<field9>9</field9>
<field10>10</field10>
<field11>11</field11>
<field12>12</field12>
<field13>13</field13>
<field14>14</field14>
</line>


Text to Word List

 Text to Word List

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

This filter takes all the incoming words and outputs them one per line, with a DOS line feed between them. This can be used to generate word lists for Indexes, encryption programs etc. Hyphenated words are recognised as single words, provided that they aren't broken across lines. To get around this limitation, use a Search and Replace filter to replace hyphens followed by line feeds with just a hyphen.

Normally you would follow this filter with a Sort and Remove Duplicates filter.


Word documents to text files

 Word documents to text files

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

This filter takes ALL incoming documents, opens them with Microsoft Word, and outputs them as text files. This can be used to process a set of Word Documents to text file format. After this filter you can add search and replace filters or any other filters you choose.

Normally you set the Output filter's Change Extension to '.TXT' to prevent the original documents being overwritten.

This filter requires Microsoft Word 98 or higher to be installed. If you wish to convert documents other than the default .DOC files, you may also need to install Word's conversion filters.

Unless you know that all documents being processed are Word documents (e.g. by using a wildcard of *.doc in the Files to Process tab), you should restrict this filter to only files of a certain extension.


Hex dump

 Hex dump

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

This changes the text to lines consisting of 16 bytes each. Each line has an 8 hex digit file index, 16 bytes (in hex) and the ASCII representation:

00000000 65 67 69 6E 0D 0A 20 20 20 20 20 20 61 64 64 72 egin.. addr
00000010 65 73 73 20 3A 3D 20 0D 0A 20 20 20 20 20 20 20 ess := ..
00000020 20 64 65 63 54 6F 48 65 78 53 74 72 28 20 28 66 decToHexStr( (f

This filter is very useful for identifying special characters to search and replace.

If you need to simply convert a file to hex, use the Email\Hex encoding filter.


Decimal dump

 Decimal dump

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

This changes the text to lines consisting of 10 bytes each. Each line has a 10 decimal digit file index, 10 bytes (in decimal) and the ASCII representation:

0000000000 080 108 101 097 115 101 032 102 101 101 Please fee
0000000010 108 032 102 114 101 101 032 116 111 032 l free to
0000000020 099 111 109 109 101 110 116 013 010 111 comment..o

This filter is very useful for identifying special characters to search and replace.

If you need to simply convert a file to hex, use the Email\Hex encoding filter.


Character maths

 Character maths

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

Performs a mathematical translation on the input characters. All operations are performed modulo 256.

The operations supported are:

    +
    -
    *
    div (the remainder is ignored)
    mod (the remainder after division)
    xor
    and
    or
    not
    shift left (0 inserted)
    shift right (0 inserted)
    rotate left
    rotate right

The hex and binary equivalent of the operand is displayed next to the decimal value that can be edited.

Operand

The maximum operand value is 255.


Add Filters

 Add Filters

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up
Macros
Add line numbers
Add left margin
Add footer
Add header
Add right margin
Add column of text
Add bytes
Add Text Side By Side
Add Repeating Text Side By Side
Word wrap
Pad to width
Truncate to width
Center in width
Right justify in width
Force to width
Format Specifiers

These filters add text or reformat it.

Add line numbers
Add left margin
Add right margin
Add header
Add footer
Add column of text
Add bytes
Add text side by side
Add repeating text side by side

Word wrap
Pad to width (Pro version only)
Center in width
Right justify in width
 Truncate to width (Pro version only)
Force to width (Pro version only)

To add new filters to the filter list, select one from the Filter menus.


Add line numbers

 Add line numbers

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter adds line numbers to the start of every line. Each line number is five digits (with leading zeroes inserted where necessary), followed by a space.

The starting line number and the line number increment can both be entered. The maximum value for each field is 2147483647.

Format

The format field allows the output of the number to be altered, which can be used for including static text on either side of the number or altering the appearance of the number (e.g. leading zeros, hexadecimal output etc). For more information on the allowed options, please see Format Specifiers. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy and entering special characters.

Don't Number Blank Lines

When this field is checked, blank lines do not have a line number printed on the left hand side. This can be used to make the output look cleaner and break up sections of code.

Skip Increment on Blank Lines

When this field is checked, blank lines are not counted.

Don't Number Blank Lines must be checked for this field to be enabled.


Add left margin

 Add left margin

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Adds the contents of the file or the specified text to the start of every line. Useful for adding email reply quotes eg. '> ' to every line. It can also be used to indent text. Sub filters can be used to pre-process the file or text being added.

The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering special characters and inserting macros. Special characters and macros are ONLY supported when they are entered in the text field - they will not be interpreted or parsed when a file's contents are included.

Note: An empty file or restriction has no lines, so no left margin will be added. Use an Add Footer filter instead.

See also

Add text to right margin
Add header text
Add footer text
Add column of text


Add right margin

 Add right margin

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Adds the contents of the file or the specified text to the end of every line. You can use this filter after a Pad To Width filter to put characters in the right hand margin. Sub filters can be used to pre-process the file or text being added.

The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering special characters and inserting macros. Special characters and macros are ONLY supported when they are entered in the text field - they will not be interpreted or parsed when a file's contents are included.

Note: An empty file or restriction has no lines, so no right margin will be added. Use an Add Header filter instead.

See also

Add text to left margin
Add header text
Add footer text
Add column of text


Add header

 Add header

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Adds the contents of the file or the specified text to the start of each file. Useful for adding advertising banners to web pages or other forms of file header. This filter is usually positioned at the top of the filter list ie just after <input>. Sub filters can be used to pre-process the file or text being added.

Note: If you wish to add a header to each page of a file that contains many pages, the best method of doing so is to use an 'Add Header Text' filter in conjunction with a search and replace for page end characters. Search for a form feed character (\f or \012), and when you find it, replace with a \f followed by your page header text.

The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering special characters and inserting macros. Special characters and macros are ONLY supported when they are entered in the text field - they will not be interpreted or parsed when a file's contents are included.

See also

Add text to left margin
Add text to right margin
Add footer text
Add column of text


Add footer

 Add footer

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Adds the contents of the file or the specified text to the end of each file. This filter is usually positioned at the bottom of the filter list ie just before <output>. Sub filters can be used to pre-process the file or text being added.

Note: If you wish to add a footer to the end of each page of a file that contains many pages, the best method of doing so is to use an 'Add Footer Text' filter in conjunction with a search and replace for page end characters. Search for a form feed character (\f or \012), and when you find it, replace with your page footer text followed by \f.

The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering special characters and inserting macros. Special characters and macros are ONLY supported when they are entered in the text field - they will not be interpreted or parsed when a file's contents are included.

Useful for adding copyright messages to the end of HTML files.

See also

Add text to left margin
Add text to right margin
Add header text
Add column of text


Add column of text

 Add column of text

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Inserts a new column of text. The position the text is inserted is determined by a column count. The leftmost column is column 1 – inserting in this column displaces all other text to the right. If the insert column given is 0, the text is inserted at the end of the line. If the insert column is negative, the text is inserted at the given position relative to the end of the line. If the insert column given is before the start of the line, or beyond the end of the line, then the text is pre-pended or appended to the line respectively. Sub filters can be used to pre-process the file or text being inserted.

The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering special characters and inserting macros.

See also

Add text to left margin
Add text to right margin
Add header text
Add footer text


Add Text Side By Side

 Add Text Side By Side

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter is used to join two files together as columns side by side. If File A represents each file from the Files to Process list, and File B is the file being added side by side, then each line of File B gets added to the right hand end of each line from File A. If File A has more lines than File B, then the lines at the end of File A won't have anything from File B (if you need them to, either make sure that both files have the same number of lines, or use the Add Repeating Text Side By Side filter). If File B has more lines than File A, then the extra lines from File B will be omitted.

You may wish to precede the Add Text Side By Side filter with a Pad to Width filter to make sure that the right hand margin of File A is not ragged when the lines are joined.

Sub filters can be used to pre-process the file or text being added.

See Also

Add Repeating Text Side By Side
Pad to Width


Add bytes

 Add bytes

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Inserts bytes at a given offset, from 0 up to the size of the file. Sub filters can be used to pre-process the file or text being inserted.

The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering special characters and inserting macros.

See also

Add text to left margin
Add text to right margin
Add header text
Add footer text


Add Repeating Text Side By Side

 Add Repeating Text Side By Side

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter is used to join two files together as columns side by side. If File A represents each file from the Files to Process list, and File B is the file being added side by side, then each line of File B gets added to the right hand end of each line from File A. If File A has more lines than File B, then File B is repeated. If File B has more lines than File A, then the extra lines from File B will be omitted.

You may wish to precede the Add Repeating Text Side By Side filter with a Pad to Width filter to make sure that the right hand margin of File A is not ragged when the lines are joined.

Sub filters can be used to pre-process the file or text being added.

See Also

Add Text Side By Side
Pad to Width


Word wrap

 Word wrap

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter reformats text that is wider than the specified column width. Use it when text has been formatted without line feeds.

The line feed added by this filter is a DOS CR/LF pair, so if you need a Unix or Mac end of line character you will need to follow it with a Convert End of Lines filter.

If you need to add line feeds to a document with fixed record sizes, use a Convert End of Lines filter.


Pad to width

 Pad to width

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

This filter ensures that all lines are of a uniform length by adding spaces to lines that are shorter than the given width.

See also

Center in width
Right justify in width


Truncate to width

 Truncate to width

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

This filter truncates lines that are longer than the specified width.

See also

Center in width
Right justify in width


Center in width

 Center in width

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter centres text in a line of the given width by adding spaces at the start and end of the line. Lines longer than this width are ignored.

See also

Pad to width
Right justify in width


Right justify in width

 Right justify in width

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter right justifies text in a line of the given width by adding spaces at the start of the line. Lines longer than this width are ignored.

See also

Pad to width
Center in width


Force to width

 Force to width

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

This filter truncates lines that are longer than the specified width.

See also

Center in width
Right justify in width


Remove Filters

 Remove Filters

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up
Remove lines from list
Remove Blanks From Start Of Line
Remove Blanks From End Of Line
Remove blank lines
Remove multiple whitespace
Remove HTML and SGML
Remove backspaces
Resolve backspaces
Remove matching/non-matching lines
Remove lines from list
Remove lines matching pattern list
Remove duplicate lines
Remove Unicode BOM (Byte Order Mark)
Remove head / tail
Remove columns
Remove delimited fields (CSV, Tab, Pipe etc)
Remove binary characters
Remove byte range
Remove line range
Remove ANSI codes
Remove all

Found under the "Remove" menu, these filters remove unwanted information.

    Remove Blanks From Start Of Line
    Remove Blanks From End Of Line
    Remove blank lines
    Remove multiple whitespace

    Remove HTML and XML\Remove all tags
    Remove HTML and XML\Remove elements (e.g. <table...>
    Remove HTML and XML\Remove attributes (e.g. height="...")
    Remove HTML and XML\Remove tag pair (e.g. <script>...</script>)

    Remove backspaces
    Resolve backspaces
    Remove line range
    Remove lines matching
    Remove lines not matching
    Remove lines from list (Pro version only)
     Remove lines matching pattern list (Pro version only)

    Remove duplicate lines
     Remove start or end of file (Pro version only)
    Remove columns (Pro version only)
    Remove delimited fields (CSV, Tab, Pipe etc) (Pro version only)
    Remove byte range (Pro version only)

    Remove binary characters (Pro version only)
    Remove ANSI codes (Pro version only)
    Remove Unicode BOM (Byte Order Mark)

    Remove all


Remove Blanks From Start Of Line

 Remove Blanks From Start Of Line

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Removes spaces and tabs from the start of every line. This is equivalant to an unindent action.

See also

Remove Blanks From End Of Line
Remove blank lines
Remove multiple whitespace
 Remove menu


Remove Blanks From End Of Line

 Remove Blanks From End Of Line

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Removes spaces and tabs from the end of every line. In general, these characters are never used and can be removed with no ill-effects.

See also

Remove Blanks From Start Of Line
Remove blank lines
Remove multiple whitespace
 Remove menu


Remove blank lines

 Remove blank lines

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Removes blank lines. Note, lines with spaces or tabs are not removed. Use the Remove Blanks From Start Of Line filters first to rectify this.

See also

Remove Blanks From Start Of Line
Remove Blanks From End Of Line
Remove multiple whitespace
 Remove menu


Remove multiple whitespace

 Remove multiple whitespace

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Removes sequences of multiple spaces or tabs and replaces them with a single space. Useful for making documents look professional – multiple spaces look untidy.

See also

Remove Blanks From Start Of Line
Remove Blanks From End Of Line
Remove blank lines
 Remove menu


Remove HTML and XML

 Remove HTML and SGML

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Use this filter to convert HTML documents to a readable format.

This filter removes HTML and XML markup tags i.e. everything including and between <> brackets.

If your file still has HTML special characters in it such as

&nbsp;
&#174;

etc, follow this filter with the included filter "html\HTML to text.fll" filter. This filter will replace the escaped special characters e.g. &#174; with (tm).

See also

Remove Blanks From Start Of Line
Remove Blanks From End Of Line
Remove blank lines
 Remove menu


Remove backspaces

 Remove backspaces

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Remove backspaces ie all ASCII code 8s.

See also

Resolve backspaces
 Remove menu


Resolve backspaces

 Resolve backspaces

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Resolve backspaces ie remove both the backspaces and the characters prior to the backspaces that would have been deleted.

See also

Remove backspaces
 Remove menu


Remove matching/non-matching lines

 Remove matching/non-matching lines

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

TextPipe will remove any lines that match/don't match the given pattern. For more options, please see the Restrict to matching lines filter. Unlike that filter, this filter is not allowed to have any sub filters.

See also

Retain matching lines (grep)/non-matching lines (inverse grep)
Retain lines found in file
Retain lines matching pattern list
Remove menu
Perl pattern matching reference
EasyPattern reference


Remove lines from list

 Remove lines from list

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

TextPipe will remove all lines that EXACTLY match any of the lines in the given text file, or in the text memo area (line breaks can be of any format).

See also

Remove lines matching pattern list
Retain lines found in file
Remove lines matching
Remove menu


Remove line range

 Remove line range

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter removes a range of lines.

Several different means of specifying the lines to remove are provided:

  1. Start line, End line. This removes all lines including and between the specified lines.
  2. Start line, width. Removes Width lines starting from (and including) line Start.
  3. End line, width. Removes Width lines backwards starting from (and including) line End.
  4. Start line to end of line. Removes all lines from the Start line to the very end of the line.
  5. Width to end of line. Removes Width lines backwards starting from (and including) the last line.

See also

Retain lines found in file
Retain lines matching pattern list
Remove menu


Remove lines matching pattern list

 Remove lines matching pattern list

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

TextPipe will remove all lines that match any of the patterns in the given text file (or specified in the text memo area). Each pattern must be specified on a line of its own (line breaks can be of any format). Each line is interpreted as a perl-style pattern, with options of case-insensitive matching, non-greedy matching, comments not allowed. If you need to change these options, precede each line by the following codes:

Change option command | Result
(?-i) | Case-sensitive matching
(?x) | Comments allowed
(?U) | Greedy matching

See also

Retain lines found in list
Retain matching lines from list
Remove menu


Remove duplicate lines

 Remove duplicate lines

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Removes successive duplicate lines. The duplicate lines must be adjacent for this filter to remove them. This filter is most commonly used after a sort - where duplicate lines will be placed together. From a set of duplicate lines, the line that gets output is the last duplicate line of the set, unless the set starts on the first line of the file, in which case the first line gets output.

Ignore case

If ignore case is checked, lines do not need to be cased identically to be considered duplicates. Two identical lines, one in upper case, and one in lower case, would be considered duplicates and removed by this filter. If ignore case is unchecked, the lines must be identical to be considered duplicates. The case checking routines are ANSI aware, so their behaviour may change depending on your locale.

Start column

The Start Column field allows you to ignore leading characters before a comparison is made. This can be very useful for ignoring line numbers is source code etc.

It can also be used (in conjunction with an external sort program) to remove duplicates from an unsorted file. First add line numbers to every line. Then use an external sort program on the file, ignoring the line numbers (the first 6 columns). Then use this filter (with Ignore Leading Characters set to 6) to remove duplicate lines. Finally, remove the line numbers using the Remove Columns (remove first 6 columns) filter.

Length

The comparison can also ignore trailing characters if desired, by setting the length to less than the length of the line.

See also

Count duplicate lines
 Remove menu


Remove head / tail

 Remove head / tail

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

This filter is used to include or exclude either lines or bytes from the start or end of a file. It performs an extension of the functionality of the Unix head and tail commands.

Include or Exclude

Specifies whether to keep or remove the selected text. Normally you know how much text you wish to include or exclude eg 10 lines.

Counting Units

Specifies the quantity of text and how it is to be counted, in lines or bytes.

Where to Count From

Start Of File and From End Of File - specify where the text should be found – at the start or end of the processed text.

See also

Remove menu


Remove columns

 Remove columns

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter is used to remove columns of text, given a column specification that describes the position of the column relative to the start or end of the line, and the width of the column.

Several different means of specifying the columns to remove are provided:

  1. Start column, End column. This removes all text including and between the specified columns. Useful for removing column in fixed width data files.
  2. Start column, width. Removes Width characters starting from (and including) column Start.
  3. End column, width. Removes Width characters backwards starting from (and including) column End.
  4. Start column to end of line. Removes all characters from the Start column to the very end of the line. Useful for making a file a uniform width.
  5. Width to end of line. Removes Width characters backwards starting from (and including) the last column.

Note - if you are removing more than one column range, it is easiest to remove ranges from right-to-left so that the position of the columns doesn't change.

See also

Remove menu
Remove columns
Remove delimited fields (CSV, Tab, Pipe etc)


Remove Unicode BOM (Byte Order Mark)

 Remove Unicode BOM (Byte Order Mark)

Submit feedback on this topic

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter removes the Unicode Byte Order Mark from the start of Unicode files, if it is present.

Bytes removed | Description
00 00 FE FF | UTF-32, big-endian
FF FE 00 00 | UTF-32, little-endian
FE FF | UTF-16, big-endian
FF FE | UTF-16, little-endian
EF BB BF | UTF-8

See also

Swap UTF-16 word order
Swap UTF-32 word order
Make Big Endian
Make Little Endian
Unicode conversion

Remove menu
Remove columns
Remove delimited fields (CSV, Tab, Pipe etc)


Remove binary characters

 Remove binary characters

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

Removes binary characters such as those higher than ASCII code 127, and those less than ASCII code 32 except for carriage returns (ASCII code 13) and line feeds (ASCII code 10).

This filter if very useful if you have a corrupted word processor file, or if you just want to see what text, if any, is inside a file. The binary information required by the word processor can be removed, leaving you with just the text.

If you wish to retain foreign characters, create a new map based on the Remove Binary filter, then customize the map to retain the characters you need.

See also

Remove menu


Remove delimited fields (CSV, Tab, Pipe etc)

 Remove delimited fields (CSV, Tab, Pipe etc)

Submit feedback on this topic

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter is used to remove fields delimited by a given character.

Choose the delimiter that separates your fields

You can choose a predefined delimiter character, or select your own. The trailing delimiter (if any) is also removed. When Comma (.csv) is chosen, TextPipe automatically handles single and double quoted strings, with embedded line feeds.

First Row Contains Field Names

If the first line of the file contains Field Names, check this box so that TextPipe can count how many fields are expected. It can then determine if a field has embedded CR/LF characters and spans multiple lines. TextPipe can also determine this without a header if the fields are properly double-quoted - TextPipe will notice the missing double quote and continue reading the record from the following line.

Remove Fields

Several different means of specifying the fields to remove are provided:

  1. Start field, end field. This removes all text including and between the specified fields.
  2. Start field, width. Removes Width fields starting from (and including) field Start.
  3. End field, width. Removes Width fields backwards starting from (and including) field End.
  4. Start field to end of line. Removes all fields from the Start field to the very end of the line.
  5. Width to end of line. Removes Width fields backwards starting from (and including) the last field.

Note - if you are removing more than one field range, it is easiest to remove ranges from right-to-left so that the position of the fields doesn't change.

See also

Remove menu
Remove columns


Remove ANSI codes

 Remove ANSI codes

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

ANSI (American National Standards Institute) codes are included in various streams of information (most commonly BBS connections), to provide a remote computer with control over cursor positioning, text attributes, etc. They are also used in connections between minicomputers and mainframe computers and the terminals connected to them.

The need to use an ANSI filter can be recognized when something like the following example shows up in a file viewed in a text editor:

<[0;1;4mas<[m - MC88000 assembler

In this example the "as" near the beginning is displayed in a different color than the rest of the line when the ANSI codes are properly processed. The Escape (ASCII 27) codes above have been replaced by the < symbol to make this line printable.

The Remove ANSI Escape Sequences filter can be used to filter out these codes and "clean up" the text so that it can be used in standard fashions such as copying and pasting into a word processor. On Unix machines the man (manual) help utility will only allow page-by-page browsing through a file in a forward direction. By piping the man output to a text file, transferring it to a DOS machine, and running it through the Remove ANSI Escape Sequences filter (and the Convert EOL filter - Unix to DOS if desired), a standard DOS editor can be used for browsing through the file, quoting from it, etc.

This filter can be enabled on the command line by using the /DA option.

See also

Remove menu


Remove all

 Remove all

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter removes all text.

It is useful in two main situations

  1. Inside a subfilter, it prevents any of the subfiltered text from re-entering the text stream. So you could restrict to lines matching a pattern, output the matching lines to a new file, and then remove them.
  2. To remove all of the text of a file, then use an Add Header filter with the @fullInputFilename macro to obtain the name of the file.

See also

Remove menu


Email Filters

 Email Filters

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up
Extract email addresses
Extract URLs
Remove email headers
Unscramble
Hex encoding
Hex decoding
Mime Base 64 encode
Mime Base 64 decode
Mime quoted printable encode
Mime quoted printable decode
UUEncode
UUDecode
XXEncode
XXDecode
HTTP Encode
HTTP Decode

(Pro version only)

Extract email addresses
Extract URLs
Remove email headers
Unscramble
Hex encoding
Hex decoding
Mime Base 64 encode
Mime Base 64 decode
Mime quoted printable encode
Mime quoted printable decode
UUEncode
UUDecode
XXEncode
XXDecode
HTTP Encode
HTTP Decode


Extract email addresses

 Extract email addresses

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

Extract email addresses. This filter searches for email addresses of the form person@server.domain, and writes them out one per line (using a DOS line feed, CR/LF).

Usually this filter is followed by a Sort filter, with the remove duplicate lines option. This is followed by a Search and Replace filter, searching for \013\010 and replacing with a comma or semi-colon, depending on the email address separator used by your email software.

See also

Remove email headers
Extract URLs


Extract URLs

 Extract URLs

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

Extract URLs. This filter lists http://, https://, ftp:// and gopher:// URLs one per line (using a DOS line feed, CR/LF).

Usually this filter is followed by a Sort filter, with the remove duplicate lines option.

See also

Extract email addresses
Remove email headers


Remove email headers

 Remove email headers

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

This filter removes the email headers that accompany emails exported to a text format. The email headers are the lines such as To:, From:, Subject: and various other message headers added by all the servers through which your email passes before it gets to its destination.

See also

Extract email addresses
Extract URLs


Unscramble

 Unscramble

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

This is a simple email encoding usually used to disguise text that some people may find offensive. The encoding is totally reversible (applying it twice removes the encoding). Only alpha characters are affected (A..Z and a..z).

Text that has been processed with the ROT13 filter usually looks like the following:

Vs lbh obgurerq gb qrpbqr guvf, lbh ernyyl unir gbb zhpu gvzr ba
lbhe unaqf!


Hex encoding

 Hex encoding

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

A very simple encoding of a file. Usually used for small files, because it wastes a large amount of space. The benefit is that the file is very easy to encode/decode, and the file cannot be corrupted passing through mail gateways.

Text that has been processed with the Hex encoding filter usually looks like the following:

496620796F7520626F74686572656420746F206465636F646520746869732C20796F75207265
616C6C79206861766520746F6F206D7563682074696D65206F6E0D0A796F75722068616E6473
21

See also

Hex decoding


Hex decoding

 Hex decoding

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

Converts a file from its hex representation back to binary.

The file to be decoded MUST NOT have any extra characters at the start or end if it is to be successfully processed.

See also

Hex encoding


Mime Base 64 encode

 Mime Base 64 encode

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

Used for binary data. Files of this type are usually given an extension of .b64.

Text that has been processed with the Mime Base 64 filter usually looks like the following:

SWYgeW91IGJvdGhlcmVkIHRvIGRlY29kZSB0aGlzLCB5b3UgcmVhbGx5IGhhdmUgdG9vIG11Y2gg
dGltZSBvbg0KeW91ciBoYW5kcyENCg==

See also

Mime quoted printable encode
Mime quoted printable decode
Mime Base 64 decode


Mime Base 64 decode

 Mime Base 64 decode

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

Used for binary data. Files of this type are usually given an extension of .b64.

The file to be decoded MUST NOT have any extra characters at the start or end if it is to be successfully processed.

See also

Mime quoted printable encode
Mime quoted printable decode
Mime Base 64 encode


Mime quoted printable encode

 Mime quoted printable encode

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

Quoted printable is used for text that is mainly readable, but may contain special characters with accents etc.

See also

Mime quoted printable decode
Mime Base 64 encode
Mime Base 64 decode


Mime quoted printable decode

 Mime quoted printable decode

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

The inverse of the above encoding.

See also

Mime quoted printable encode
Mime Base 64 encode
Mime Base 64 decode


UUEncode

 UUEncode

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

Usually used for transmitting binary files inside an email. Files of this type are usually given an extension of .uue. Warning – UUencoded text may be corrupted when passing over a mainframe mail gateway. To avoid corruption, use Mime Base 64 or XXEncode.

UUEncoding's major drawback is that there are many flavors of uuencode in existence and this plays against interoperability. Gateways that convert between EBCDIC and ASCII, furthermore, may damage some of the characters used in the uuencode format. The base64 encoding is designed to be invulnerable to all known gateways. The UUencoding implemented by TextPipe uses the following characters:

`!#$%&''()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_
+-0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz

Text that has been processed with the UUEncode filter usually looks like the following:

UUEncoded attachment by TextPipe Evaluation Edition
begin 0777 clipboard
M268@>6]U(&)O=&AE<F5D('1O(&1E8V]D92!T:&ES+"!Y;W4@<F5A;&QY(&AA
B=F4@=&]O(&UU8V@@=&EM92!O;@T*>6]U<B!H86YD<R$-"@``
`
end

See also

UUDecode
XXEncode
XXDecode


UUDecode

 UUDecode

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

Mail attachments can be uuencoded, use this filter to convert the file back to its correct form. Files of this type are usually given an extension of .uue.

UUEncoding's major drawback is that there are many flavors of uuencode in existence and this plays against interoperability. Gateways that convert between EBCDIC and ASCII, furthermore, may damage some of the characters used in the uuencode format. The base64 encoding is designed to be invulnerable to all known gateways. The UUencoding implemented by TextPipe uses the following characters:

`!#$%&''()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_

See also

UUEncode
XXEncode
XXDecode


XXEncode

 XXEncode

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

Essentially identical to UUEncode except that the character set used is different to allow it to pass through EBCDIC gateways without corruption.

The XXencoding implemented by TextPipe uses the following characters:

+-0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz

Text that has been processed with the XXEncode filter usually looks like the following:

XXEncoded attachment by TextPipe Evaluation Edition
begin 0777 clipboard
hGKMUSKxp647jR4VZQaJY65Fj64FZMqxYNG-oO4Zn90-tPrIUQaJVP4lt64VV
WRaIUR4xj64ppMqUUR4ZhNG-jPUo8SKxpQW-cMKtYQm2B0U++
+
end

See also

UUEncode
UUDecode
XXDecode


XXDecode

 XXDecode

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

Essentially identical to UUDecode except that the character set used is different to allow it to pass through EBCDIC gateways without corruption.

The XXencoding implemented by TextPipe uses the following characters:

+-0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz

See also

UUEncode
UUDecode
XXEncode


HTTP Encode

 HTTP Encode

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

This filter is used to encode text for use in an HTTP header – a (usually) small piece of text that accompanies a web page request to a web server. This filter is very useful for debugging CGI scripts because it can create HTTP requests in the correct form. HTTP encoded text usually looks like the following:

a+%28usually%29+small+piece+of+text+that+accompanies+a+web+page+request+to+a+web+server.+This+filter+is+very+

See also

HTTP Decode


HTTP Decode

 HTTP Decode

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

This filter is used to decode text from an HTTP header – a (usually) small piece of text that accompanies a web page request to a web server.

See also

HTTP Decode


Search and replace filters

 Search and replace

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up
Exact Search and replace
EasyPattern Search and replace
Perl Pattern Search and Replace
Unicode Search and replace (UTF-16LE)
Unicode Pattern Search and replace (UTF-16LE)
Sounds-like Search and Replace
Edit Distance Search and Replace
Old Pattern (egrep) Search and Replace
Brief Pattern Search and Replace
MS Word Pattern Search and Replace
Search/replace lists
Import a search/replace list from Excel, CSV or Tab
Converting Search/Replaces from other sources
Prompt On Replace

TextPipe offers a number of different search and replace types:

  • Exact match - the find string is matched exactly
  • EasyPattern - an easy to use pattern matching language
  • Pattern - the find string is a perl-style pattern match expression

  • Sounds like - the find string is used for soundex matching. Soundex matching is a method that makes surnames such as Smith, Schmidt and Smythe all appear identical. It is very useful for finding misspellings of names

  • Edit distance - Determines the number of 'edits' between the search word and words in the text being processed
  • Old Pattern (egrep) - the find string is an egrep-style pattern match expression, which is not as powerful as perl pattern matching
  • Brief pattern match - the find string is a pattern match expression in the style of the text editor Brief
  • MS Word pattern - the find string is a pattern match expression in the style of Microsoft Word
  • Unicode exact match - used for finding text in Unicode files
  • Unicode pattern match - for finding Unicode text patterns in Unicode files
  • Load a search/replace list from Excel, CSV or Tab
  • Search/replace list
  • Find an HTML/XML element (e.g. <table...>)
  • Find an HTML/XML attribute (e.g. height="...")
  • Find between an HTML/XML pair (e.g. <script>...</script>)

See also

  • An Introduction to Pattern Matching

  • Example patterns
  • Pattern matching reference
  • EasyPattern pattern matching
  • EasyPattern reference

  • Unicode patterns reference


HTML/XML Restriction Filter

 Restrict to tag/attribute

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter restricts sub filters to operate on just the text within the given tag. Normally this filter has a Restrict to attribute sub filter. If your tag is not in the list, just type it in.

Note:

  • This filter copes with single and double quotes, and escaped quotes within them
  • This filter copes with angle brackets within single and double quotes
  • <!-- --> comment tags are handled specially. Comment tags can have ANY text inside them - quotes are not matched
  • <!DOCTYPE > tags are handled specially to allow for included DTDs enclosed in []
  • <!CDATA[ ... ]]> tags are handled specially to allow for any text within the square brackets.

Include the entire tag

When checked, the entire tag is passed to the sub filter, not just the text after the tag name. Normally this is not required as the existing tag is being edited. If you want to remove the entire tag, then check this field, and add a Remove All sub filter.

Match case

When checked, the tag name must match the case of the name exactly. Normally this is checked for XML, which is case-sensitive, and unchecked for HTML, which is not case-sensitive.

Restrict to attribute

This filter restricts sub filters to operate on just the text within the given tag. Normally this filter is inside a Restrict to tag filter e.g. 'Restrict to tag <table...>', sub filter of 'Restrict to attribute height="..."', sub filter of search and replace. If your attribute is not in the list, just type it in.

Include the quotes

When checked, the surrounding single or double quotes (if any) are also sent to the sub filter. Normally they are not required as the existing value is being edited. If you want to remove the entire attribute, then check this field and the next field, and add a Remove All sub filter.

Include the attribute name

When checked, the actual attribute name and the equals sign are also sent to the sub filter. Normally they are not required as the existing value is being edited. If you want to remove the entire attribute, then check this field and the previous field, and add a Remove All sub filter.

Restrict to between tags

This filter restricts sub filters to operate on just the text between the given tag pairs (which need not match). If your tag is not in the list, just type it in.

Include the start and end tag

When checked, the actual start and end tags are also sent to the sub filter. Normally they are not required as the existing value is being edited. If you want to remove the start and end tag and all text between them, then check this field, and add a Remove All sub filter.

See also

Remove duplicate lines
 Count duplicates
 Randomize lines
Restriction filters


Search/replace lists

 Search/replace lists

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter is used to easily define a set of search/replace pairs that have identical search options.

From file

Loads a series of search/replace pairs from a file dynamically at run time. If the file extension is .XLS, the file is assumed to be Excel format, if the extension is .TAB the file is assumed to have tab-delimited values, and any other extension (including .CSV) is assumed to have Comma-Separated Values.

Excel spreadsheet (.xls)

The Excel spreadsheet must have data in Sheet 1, and be formatted with the search term in column A, and the replacement for it in column B. No heading is allowed. The loading process stops when a blank cell is reached in column A. It might look as follows:

Tab-delimited file (.xls)

The list is loaded from a Tab-delimited file. Such a file can be created with Notepad, and looks like this:

a       b
aaaabbb cccccdddd ssss

a

a
a       b      c

CSV-delimited file (.csv or any other extension)

The list is loaded from a CSV-delimited file. Such a file can be created with Notepad, and looks like this:

a,b
"a","b"
"aaaa""bbb","ccccc""dddd""ssss"
a,
a
a,b,c
"a","b","c"
"aaaa""bbb","ccccc""dddd""ssss","sssss""ssss"

Using these values

Use the grid to specify multiple search/replace pairs.

The string, pattern or Sounds like text to find. The string can contain unprintable characters. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering special characters and inserting pattern match characters.

Find/Replace options

Find type, Match case etc, as for normal search/replace.


Import a search/replace list from Excel, CSV or Tab

 Import a search/replace list from Excel, CSV or Tab

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This series of menu options allows you to import a list or table of search and replace pairs from an external file. The type of search and replace added, and the options that are set, are predetermined by the last search and replace filter selected. So to set the default options, you must initially select a search and replace filter with the desired options. This can also be done programmatically using the COM interface, or search and replace pairs can be added via the command line using pairs of /S= and /R= options. You can also use a Search/Replace list filter to load a file dynamically.

Import list from Excel

The Excel spreadsheet must have data in Sheet 1, and be formatted with the search term in column A, and the replacement for it in column B. No heading is allowed. The loading process stops when a blank cell is reached in column A. It might look as follows:

Import list from CSV-delimited file

The list is loaded from a CSV-delimited file. Such a file can be created with Notepad, and looks like this:

a,b
"a","b"
"aaaa""bbb","ccccc""dddd""ssss"
a,
a
a,b,c
"a","b","c"
"aaaa""bbb","ccccc""dddd""ssss","sssss""ssss"

Import list from Tab-delimited file

The list is loaded from a Tab-delimited file. Such a file can be created with Notepad, and looks like this:

a       b
aaaabbb cccccdddd ssss

a

a
a       b      c


Special Filters

 Special Filters

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up
Extract Matches
Database connection
Sort
Run external filter
Script Filter
Comments
Display Debug Window
Split files
Merge files
Count duplicate lines
Retain matching/non-matching lines
Retain lines matching pattern list
Move columns
Move fields (CSV, Tab, Pipe etc)
Copy columns
Copy fields (CSV, Tab, Pipe etc)
Repeat file's contents
Randomize lines
Reverse line order
Reverse line order
XSLT Filter

Found under the "Special" menu, these filters provide specialized processing suited to a variety of tasks.

    Extract matches (Pro version only)
    Database connection     (Pro version only)
    Sort
    Run external program (Pro version only)
    Script (Pro version only)
    Comment
    Debug filter
    XSLT filter

     Split files
    Merge (join) files
    Count duplicate lines

    Retain matching lines (grep)
    Retain non-matching lines (inverse grep)
    Retain lines from list (Pro version only)
    Retain lines matching pattern list (Pro version only)

    Move columns (Pro version only)
    Move fields (CSV, Tab, Pipe etc) (Pro version only)

    Copy columns (Pro version only)
    Copy fields (CSV, Tab, Pipe etc) (Pro version only)

    Repeat file contents
    Randomize lines
    Reverse each line
    Reverse line order


 Extract Matches

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

This filter only keeps text (not lines) that match a regular expression. All other text is discarded. If you do not need this filters ability to also include the current filename and line number, you should use the Extract option of the Find and Replace filter because it is far more flexible.

Extract Matches

The string or pattern to find. The string can be found anywhere on a line or over multiple lines. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering special characters and inserting pattern match characters.

Ignore Case

When checked, the case of the string found doesn't have to be identical to the Match Pattern exactly.

Output Count of Matches

When checked, outputs a count of the number of matches in each file.

Include Filename

When checked, outputs the filename prior to showing its matches.

See also

Retain matching lines
Remove matching lines


Database connection

 Database connection

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

The database connection filter connects to an OLE DB/ODBC database and retrieves data. It assumes that its input text is a sequence of SQL statements that can be run against the specified database. The data (if any) returned after executing each statement is formatted according to the output style. The filter assumes that the input text is a series of semi-colon (;) terminated SQL statements. Semi-colons that are inside single or double quotes are ignored.

The data output can then be manipulated by subsequent filters to

  • generate a database extract (which can be converted, modified or reformatted)
  • update data from the original table after performing name and address cleansing (requires a subsequent database connection filter)
  • insert data into a new table (requires a subsequent database connection filter)

Sample extracting from a database

Here's a quick example of how to use a database filter to extract data from a database:

  1. Add a Special Menu\Database filter
  2. Configure it to point to your database using the Build button e.g. to the Northwind database that comes with MS Access
  3. Set the database filter output options, e.g. for Delimited Output
  4. In the Trial Run area, type

    select * from customers

    or use

    select * from <your table name>

    For an Excel spreadsheet (first setup an ODBC connection to point to the Excel file, then point the TextPipe database filter at the ODBC connection), use

    SELECT * FROM [Sheet1$]
  5. Click the [Trial Run] button, to see data extracted and populated in the Trial Run Output area. You can also set the Input Filter to use the Trial Run area by default, and then just click the [Go] button. To save this data to a file, set the Single File Output option of the Output Filter.

Database connection string

The database connection string gives TextPipe the name of the database and tells it how to connect to it. We recommend using the Build button unless you are familiar with setting the connection string manually.

Build

The build button allows the connection information to be constructed. A username and password can be specified in the database connection string if required.

Command Timeout

The command timeout (in seconds), allows you to set the time after which a query will be automatically failed. A setting of 30 will abort a query with an error message if it runs for longer than 30 seconds (the default). The command timeout setting is not implemented by all database drivers. The maximum value is 2147483647.

Output Style

The output style determines how the data will be output:

Output header row/DTD

When enabled, an XML document type definition (DTD) is output for XML files, or a header line with column titles is output for Delimited or Fixed width output formats. TextPipe forces all column heading to be unique, and to conform to XML naming conventions without any special characters. If TextPipe doesn't automatically generate the header you need to can easily add your own using an Add Header filter.

Delimited

The field delimiter appears between each column of data, and the text qualifier appears at the start and end of string or character type fields. If the string field contains the delimiter character it is escaped by repeating the delimiter character. A tab character (\t or \009) may be used to separate columns to generate tab delimited data. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy and entering special characters. If Output Header Row/DTD is checked TextPipe automatically generates a Header Row with column names (shown in italics at the start of the example below).

"company","productid"
"Dushkin/MCgraw-Hill",100503
"Programma Electric AB",131686
"Ohio University",131686
"DSL Shops",131686

Fixed

The data is output left justified in a column width determined by the database (specific to each field). If the column width is greater than 4096 then the field is output with no padding. If Output Header Row/DTD is checked TextPipe automatically generates a Header Row with column names (shown in italics at the start of the example below).

company                    productid
Dushkin/MCgraw-Hill        100503
Programma Electric AB      131686
Ohio University            131686
DSL Shops                  131686

XML

TextPipe outputs the data in XML format, with each column output an an entity. All XML-special characters are escaped. If Output Header Row/DTD is checked TextPipe automatically generates an XML schema (DTD) for the data being extracted (shown in italics at the start and end of the example below). You can use an Add Header filter to add your own DTD at the start of the file.

<?xml version="1.0" standalone="yes" ?>

  <!-- generator="TextPipe Search/Replace" -->
  <!-- companyURL="www.crystalsoftware.com.au" -->
  <!-- SQLquery="select company,productid from shareit" -->

  <!-- event="DTD start" -->
  <!DOCTYPE TextPipeRows [
    <!ELEMENT TextPipeRows (TextPipeRow*)>
    <!ELEMENT TextPipeRow (company, productid)>
    <!ELEMENT company (#PCDATA)>
    <!ELEMENT productid (#PCDATA)>
  ]>
  <!-- event="DTD end" -->

  <TextPipeRows>

  <TextPipeRow>
    <company>Dushkin/MCgraw-Hill</company>
    <productid>100503</productid>
  </TextPipeRow>

  <TextPipeRow>
    <company>Programma Electric AB</company>
    <productid>131686</productid>
  </TextPipeRow>

  <TextPipeRow>
    <company>Ohio University</company>
    <productid>131686</productid>
  </TextPipeRow>

  <TextPipeRow>
    <company>DSL Shops</company>
    <productid>131686</productid>
  </TextPipeRow>

</TextPipeRows>

  <!-- rowCount="4" -->

Insert script for table <tablename>

With this mode setting TextPipe will generate a SQL Insert Script that can be applied to any database. You must specify the name of an existing table to insert the data into. If Output Header Row/DTD is checked, TextPipe includes the column names to insert into (shown in italics in the example below). You can use an Add Header filter to add a drop/create table statement at the start of the script.

insert into existing_table ( company, productid ) values ( "Dushkin/MCgraw-Hill", 100503 );
insert into existing_table ( company, productid ) values ( "Programma Electric AB", 131686 );
insert into existing_table ( company, productid ) values ( "Ohio University", 131686 );
insert into existing_table ( company, productid ) values ( "DSL Shops", 131686 );

BLOB data

If your query returns BLOB (Binary Large Object) data, a sub filter can be used to encode the binary data. The most common method to use is a mime base64 encoding.

Errors

If you have difficulties using this filter, you may need to update the database drivers and the corresponding system components on your system. Click here to download the Microsoft MDAC (Data Access Components) library.


Sort

 Sort

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Arranges the input lines into sequence.

Sort Type

The sort type controls the method by which items are sorted. The available options are:

  • ANSI sort (case insensitive)
  • ANSI sort (case sensitive)
  • ASCII sort (case insensitive)
  • ASCII sort (case sensitive)
  • Numeric sort
  • Sort by length of line

An ANSI sort is locale-specific (ie can vary from country to country). Lower case letters generally have lower codes than upper case letters, hence a word will sort in lower case before the same word in upper case e.g. 'Freddo' will appear before 'FREDDO'. This is opposite to an ASCII sort where upper case words appear first.

An ASCII sort is based on the ASCII code (or binary value) of each letter in a word. Upper case letters have lower codes than lower case letters, and hence the word 'HELLO' will appear before the word 'hello'.

Both ANSI and ASCII sorting have a case-sensitive and case-insensitive sort. With case-sensitive sorts, the word 'Volkswagon' is different to the word 'volkswagon'. With a case-insensitive, the word 'BMW' is the same as the word 'bmw'.

Numeric sort allows lines to be sorted according to their numeric value. The numeric value must appear at the start of the line (leading spaces are allowed). The number must be in decimal, and can be in floating point format. Any non-numeric characters after the number are ignored. If the line has no valid numeric value it is given a value of 0. If you have dates formatted in YYYYMMDDHHMMSS order you can easily sort by date based on the value.

Sort by length of line allows lines to be sorted by their length. This can be handy for detecting lines in database extracts which are longer or shorter than they are allowed to be.

Sort by date and time allows lines to be sorted by a date and time value. The value must use the current locale’s date/time format. In the US, this is commonly MM/DD/YY HH:MM:SS format. Specifying AM or PM as part of the time is optional, as are the seconds. Use 24-hour time (7:45 PM is entered as 19:45, for example) if AM or PM is not specified.

Sort by date allows lines to be sorted by a date value. The value must consist of two or three numbers, separated by the character defined by the current's locale's Date Separator. The order for month, day, and year is determined by the current locale--possible combinations are m/d/y, d/m/y, and y/m/d. If the value contains only two numbers, it is interpreted as a date (m/d or d/m) in the current year.

Sort by time allows lines to be sorted by a time value. The value must consist of two or three numbers, separated by the current locale's Time Separator character, optionally followed by an AM or PM indicator. The numbers represent hour, minute, and (optionally) seconds, in that order. If the time is followed by AM or PM, it is assumed to be in 12-hour clock format. If no AM or PM indicator is included, the time is assumed to be in 24-hour clock format.

Remove duplicate lines

Lines with the same sort value can be kept or removed.

Note that there is no way of knowing which line from a group of lines with the same sort value will actually be kept. eg. with input of 'AAA', 'BBB', 'CCC' in a Sort by length of line, there is no way of knowing which of the 3 lines will actually appear in the output as each has a length of 3 and therefore appears to be the same.

Descending order

Arranges the output in descending order (highest value first). The default (unchecked) is to sort in ascending order (lowest value first).

Start column

The Start Column field allows you to ignore leading characters before a comparison is made. This can be very useful for ignoring line numbers is source code etc.

It can also be used (in conjunction with an external sort program) to remove duplicates from an unsorted file. First add line numbers to every line. Then use an external sort program on the file, ignoring the line numbers (the first 6 columns). Then use this filter (with Ignore Leading Characters set to 6) to remove duplicate lines. Finally, remove the line numbers using the Remove Columns (remove first 6 columns) filter.

Length

The comparison can also ignore trailing characters if desired, by setting the length to less than the length of the line.

Technical Notes

  • With the QuickSort algorithm, two lines with the same sort value that appear in order at the input may not necessarily appear in order in the output (e.g. 'AAA','aaa' in a ASCII case-insensitive sort may appear as 'aaa','AAA' at the output).
  • If you have enough memory, the entire sort is performed in memory for speed. Every 10000 lines, TextPipe checks to see if there is less than 16MB of physical memory (not virtual memory!). If so, it writes the sorted results so far to a temporary file and then continues. If the Output Filter is set to File Output or Single File Output, any temporary files are written to the same folder as the output file. If the Output Filter is set to Clipboard Output then any temporary files are written to the current folder. All temporary files are removed as the sort progresses.

See also

Remove duplicate lines
 Count duplicates
 Randomize lines


Run external filter

 Run external filter

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

This filter allows an external program to be run to complement TextPipe. Examples might include a sorting program for very large files (for files larger than main memory).

The steps that TextPipe follows for each file are:

  1. Write all data to the filename specified by "Input file for external program" (after any preceding filters have have performed their work)
  2. When the file has been completely processed, run the external program and wait for it to finish. The external program is assumed to use the filename specified by "Input file for external program" as input, and write its output to the filename specified by "Program's output file"
  3. The data in the filename specified by "Program's output file" is read in and fed through any remaining filters.

Command line

Enter the command line of the program. This must be the COMPLETE command line, including drive, folder and extension of the program, plus any required parameters.

Input file for external program

This is the file that TextPipe writes its "text processed so far" to for the external program to then read and process.

Program's output file

This is the file that the program must write its results to for TextPipe to retrieve and feed into any remaining filters.


Script Filter

 Script Filter

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up
TextPipe Internal Automation Object
Microsoft script control

(Pro version only)

This filter allows you to program your own TextPipe filter in VBScript, JScript, VBA, PerlScript, Python, Lisp, REXX etc. You could use your own code to perform custom text processing, e.g. adding line numbers to the end of each line, deleting every second line, deleting every third blank line etc. You could use a message box to ask the user for some input, and then add that input to every line. Or you could invoke Excel, paste text into it, perform a complex statistical calculation and then put the result back in the file. See the example filters in the VBScript folder (such as the one to copy numeric data into an excel spreadsheet, create a 3d graph and then rotate it!). The possibilities are endless!

You can find reference information for VBScript and JScript on the web at Microsoft's web site. This includes updates for the ActiveX Script Control that is necessary to provide these features. Other script languages include PerlScript, Python, REXX, Lisp and VBA.

Inside an script, TextPipe provides the TextPipe internal automation object so that scripts can determine vital data about the current filter, and turn on and off sub filters according to programmed criteria.

See also Automating TextPipe from another program

Script Language

The script language to use. Default values are VBScript, JScript and VBA, but you can download and install others (they will not appear in the list - you must enter them manually). TextPipe will display an error message if the language you specified is not installed. You can download the Microsoft VBScript and JScript engines and reference documentation from the Microsoft web site at http://msdn.microsoft.com/scripting/default.htm

Timeout (milliseconds)

Specify the timeout in milliseconds. A script will be aborted if it runs for more than the specified number of milliseconds. The default value is 10000 (i.e. 10 seconds).

Check script

This button checks the script for any errors. If there is an error, a dialog box will appear with a description of the error.

Use default

This button replaces the existing script with the default script for the current language. This can be used as a template for your own code. TextPipe comes with a default script for VBScript inbuilt, and a sample JScript script can be loaded from the VBScript folder.

Set default

This button sets the default script for the current language to the current script. The default script is language dependant, so you can have different scripts for JScript and VBScript etc.

Functions

The following functions are REQUIRED by the script filter. Each filter is followed by explanatory text.

A processing run results in a series of calls as follows:

startJob()

for each input file
  startFile()

  for each line of the file
    processLine(...)

  endFile()

endJob()

function processLine( line : string; EOL : string) : string

TextPipe calls this function for every line in the input file (if your file has 10,000 lines, this function will get called 10,000 times). The function must return the processed line with the EOL string already appended to it.

Lines can be of unlimited length, and are delimited by CR/LF, CR or LF, depending on how the file was generated. TextPipe allows a file to contain more than one kind of end of line character. If needed, you can force all end of line characters to be standard by preceding this filter with a convert end of lines filter. You can also use your own end of line character instead of EOL.

If you need to process multiple lines, paragraphs, pages of text or the entire file, the best approach is to accumulate each line and EOL into a global buffer variable, and then output the entire processed buffer in the endFile() function.

sub startJob()

Called once at the start of a processing job, before any files are processed. Use it to initialize any job-wide variables. A processing job is the combination of all filters applied at once.

sub endJob()

Called once at the end of a processing job, after all files have been processed. Use it to dispose of OLE objects etc. A processing job is the combination of all filters applied at once.

function startFile() : string

Called at the start of each file. Use it to initialize file variables. The return value is any string that must be output at the beginning of the file (header text).

Note: startFile() is ALSO called when the Script is a sub filter, for each text value that the sub filter operates on. The impact of this is that real per-file initialization/finalization needs to be performed in a script that is not inside any sub filters. Global variables set using TextPipe.setGlobalVar should be initialized in this script as well.

function endFile() : string

Called at the end of each file. The return value is any string that must be output at the end of the file (footer text). This may also be used to flush any pending output, such as when the entire file is buffered before being output.

Note: endFile() is ALSO called when the Script is a sub filter, for each text value that the sub filter operates on. The impact of this is that real per-file initialization/finalization needs to be performed in a script that is not inside any sub filters. Global variables set using TextPipe.setGlobalVar should be initialized in this script as well.

Global variables

You can declare global variables at the top of the script (see the 'a' variable below). You should initialize these values in the startJob() function, and dispose of them (e.g. for OLE Objects) in the endJob() function.

Debugging

Currently, the easiest way to debug the script is to use VBScript MsgBox() or alert() functions to display text.

Dim MyVar
MyVar = MsgBox ("Hello World!", 65, "MsgBox Example")
   ' MyVar contains either 1 or 2, depending on which button is clicked.

JScript: Does NOT have an alert or MsgBox method. These are part of Internet Explorer, not JScript (but strangely they are in VBScript). Use the alert function of the TextPipe object instead.

Example VBScript

'This example script keeps a line counter that
'does not reset for each file

'Our line counter
dim a

'Called for every line in the file
'EOL contains the end of line characters (Unix, DOS or Mac) that must be
'appended to each line
function processLine(line, EOL)
  a = a + 1
  'add the count to the end of each line
  processLine = line & " " & a & EOL
end function

'Called at the start of a processing job -
'perform one-time initialization here
sub startJob()
  a = 0
end sub

'Called at the end of a processing job -
'destroy any declared objects here
sub endJob()
  'do nothing
end sub

'Called before each file is opened -
'perform per-file initialization here
function startFile()
  startFile = ""
end function

'Called before each file is closed -
'flush all pending file output here
function endFile()
  endFile = ""
end function

Example JScript

//This example script keeps a line counter that
//does not reset for each file

//Our line counter
var a;

//Called for every line in the file
//EOL contains the end of line characters (Unix, DOS or Mac) that must be
//appended to each line
function processLine(line, EOL)
{
a = a + 1;
//add the count to the end of each line
return line + " " + a + EOL;
}

//Called at the start of a processing job -
//perform one-time initialisation here
function startJob()
{
a = 0;
}

//Called at the end of a processing job -
//destroy any declared objects here
function endJob()
{
; //do nothing
}

//Called before each file is opened -
//perform per-file initialisation here
function startFile()
{
return "";
}

//Called before each file is closed -
//flush all pending file output here
function endFile()
{
return "";
}


TextPipe Internal Automation Object

 TextPipe Internal Automation Object

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

The TextPipe automation object ('TextPipe') may only be called from a VBScript or JScript script to request information from TextPipe or to control processes and log information.

See also

Automating TextPipe from another program

Normal Functions

sub logError( messageText : string )

Logs an error message string to the log file and to the status window's Error tab e.g.

function processLine(line, EOL)
 processLine = line & EOL
  TextPipe.logError( line & " found " )
end function

sub logWarning( messageText : string )

Logs a warning message string to the log file e.g.

function processLine(line, EOL)
 processLine = line & EOL
  TextPipe.logWarning( line & " found " )
end function

sub logInfo( messageText : string )

Logs an information message string to the log file e.g.

function processLine(line, EOL)
 processLine = line & EOL
  TextPipe.logInfo( line & " found " )
end function

function inputFilename() : string

Returns the name (without the drive or path) of the file being processed e.g.

function processLine(line, EOL)
  processLine = TextPipe.inputFilename & " - " & line & EOL
end function

function fullInputFilename() : string

Returns the name (with the drive and path) of the file being processed.

function processLine(line, EOL)
  processLine = TextPipe.fullInputFilename & " - " & line & EOL
end function

function version() : string

Returns the TextPipe version number e.g.

function startFile()
  dim crlf
  crlf = chr(13) & chr(10)

  startFile = _
    "TextPipe version " & TextPipe.version & crlf & _
 "Input filename=" & TextPipe.inputFilename & crlf & _
 "Full input filename=" & TextPipe.fullInputFilename & crlf & _
 crlf

end function

sub terminate()

Terminates the current filter thread.

function processLine(line, EOL)
  if EOL = "" then
  TextPipe.logError( "Line with no EOL found " )
    TextPipe.terminate
  else
    processLine = line & EOL
  end if
end function

function getGlobalVar( name : string ) : string

Returns the string value of the named variable. Name is not case-sensitive, and multi-line values can be stored.

Global variables are shared between all filters in the same thread e.g.

function processLine(line, EOL)
  processLine = TextPipe.getGlobalVar( "myvariable" ) & line & EOL
end function

sub setGlobalVar( name, value : string )

Sets the value of a global variable called "name" to value. Name is not case-sensitive, and multi-line values can be stored.

Global variables are shared between all filters in the same thread e.g.

function processLine(line, EOL)
  TextPipe.setGlobalVar "myvariable", line & EOL
 'still output the text unaffected
 processLine = line & EOL
end function

sub terminateTextPipe()

Terminates TextPipe. Warning! This function may have undesirable side-affects.

sub alert( message : string )

Displays a message in a window, with an OK button to close it. This helps JScript users debug their code - as there is currently no JScript alert function as there is for VBScript.

property line : integer

Returns the current line of the text being processed. If the text is inside a restriction, this reports the current line number of the restricted text.

property offset : integer

Returns the current file offset of the text being processed. If the text is inside a restriction, this reports the offset of the restricted text.

Sub filter functions

The following functions only operate when the ActiveX script filter has sub filters.

function subFilterEntireText( inputText : string ) : string

Applies the sub filters to inputText as though inputText was an entire file in its own right. If you need more control over the sub filters use the next set of functions e.g.

function processLine(line, EOL)

  lineNumber = lineNumber + 1

  if (lineNumber mod 2) = 0 then
    'an even-numbered line
    processLine = TextPipe.subFilterEntireText(line) & EOL
  else
    'an odd-numbered line
    processLine = line & EOL
  end if

end function

sub initSubFilters()

Prepares sub filters for processing text. MUST be called before sending data into sub filters.

dim lineNumber

function processLine(line, EOL)

  lineNumber = lineNumber + 1

  if (lineNumber = 100) then
    'start the special processing on line 100
    TextPipe.initSubFilters
    TextPipe.subFilterChunk( line & EOL )
    processLine = ""
  else if (lineNumber > 100) and (lineNumber < 200) then
    'special processing for lines 100..200
    TextPipe.subFilterChunk( line & EOL )
    processLine = ""
  else if (lineNumber = 200) then
    'end special processing on line 200
    TextPipe.flushSubfilters
    processLine = line & EOL
  else
    processLine = line & EOL
  end if

end function

function endFile()

  if (lineNumber = 100) then
    endFile = ""
  else if (lineNumber > 100) and (lineNumber < 200) then
    TextPipe.flushSubfilters
    endFile = ""
  else if (lineNumber = 200) then
    TextPipe.flushSubfilters
    endFile = ""
  else
    endFile = ""
  end if

end function

function startJob()
   lineNumber = 0
end function

sub flushSubfilters()

Flushes all cached data from all sub filters prior to resuming normal straight-through processing.

sub subFilterChunk( inputString : string )

Sends inputString to the sub filters. Remember to call flushSubfilters at the end of the file or beforehand.


Comments

 Comments

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter allows you to put comments into your filter lists. This can be handy for documenting complicated search and replace sequences. Comments are also handy to group filters together that perform a common function. Simply drag and drop the filters onto the comment filter.

The top line is used as the description in the filter list.

The right-click menu on both fields supports Undo, Cut, Copy, Paste, Delete and Select All.


Display Debug Window

 Display Debug Window

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

A debug filter is very handy for debugging filters. When text is passed through this filter, it places the output into a window so that you can see what the text looks like at that stage of the filtering process.

See also

Partial Trial Runs


Split files

 Split files

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter breaks files at specific boundaries. It is very useful for chopping up large files so that they can be transferred on floppy disk, then joined together using the merge function. Files are numbered with an extension of 000, 001, 002, etc.

If the output filter is set to Output to Clipboard or Single File Output, the split still occurs but the results are merged back together by the output filter as the very last stage in the filtering process.

Split at Size of (bytes)

When selected, files are split when they reach the given size (in bytes). The maximum file size is 2147483647 (2 GB).

Common Sizes

Common disk sizes are provided - choose one from the list and it will be defaulted to the field on the left

Split at Character

When selected, files are split when the given character is found (after a specified number of occurrences). If you need to split a file at a particular string, you can use a search and replace to add a special character that is unused elsewhere in the file (e.g. a character 255), and then perform a Split At Character on character 255.

Character

The character to split on when found

After this many

Only split after this many occurrences have been found in one file. The maximum is 2147483647.

Split Position

Where to split:

  • Before char - split character is output at start of next file
  • After char - split character is output at end of first file
  • At char - split character is removed

Split After Lines

The Split occurs after the specified number of lines have occurred. This is handy for dividing print files up into separate files, with 66 lines per page. The maximum value is 2147483647.

Split Filename

If you wish, you can control the names of the output files. All names are based on the input filename, in the input filename's folder. If you wish to control the output path, you MUST use the output filter.

Macro name | Use
%f | The input file's filename e.g. 'readme'
%e | The input file's extension without any period e.g. 'txt'
%d | The split file counter, starting from 0.
%3.3d | The split file counter, starting from 0, formatted in a field 3 wide with leading zeros.
%5.5d | The split file counter, starting from 0, formatted in a field 5 wide with leading zeros.
%x | The split file counter in hexadecimal.

See also

Merge


Merge files

 Merge files

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

The merge files filter is used to join files together. The content from each file is simply appended to the content from the previous file, with no extra characters e.g. if you joined files A, B and C, the merge files filter would result in output like this:

[file A][file B][file C]

This filter can be used for:

  • Joining together binary files that have been split
  • Joining together the processed results from large numbers of smaller files e.g. extracting email addresses or URLs from a large number of web pages, then using the merge files filter to join the results together so that a sort and remove duplicates can be performed.

By contrast, the output filter joins all input files together when it is set to Output to Clipboard or Single File Output. The join is performed as the very last stage in the filtering process. A merge files filter allows the merge operation to be followed by additional text processing operations, typically a sort and remove duplicates, or adding a header or footer. If the output filter is also set to Output to Clipboard or Single File Output, the merge filter filename is overridden by the output filter. The content from all files is joined at the point that the merge filter appears in the filter list.

If you join text files together, you can ensure that each file ends properly (i.e. that the start of subsequent files are not joined to the end of a line of the previous file) by using an Add Footer filter (with \013\010) before the Merge files filter.

See also

Split
Output Filter


Count duplicate lines

 Count duplicate lines

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

The Count Duplicate Lines filter outputs a count of the number of times a line has been repeated (only if there are 2 or more occurrences). The file should be sorted prior to this filter so that duplicate lines are placed next to each other. A single copy of each duplicated line is output, with a count of the number of duplicates at the start of each line. From a set of duplicate lines, the line that gets output is the last duplicate line of the set, unless the set starts on the first line of the file, in which case the first line gets output.

Ignore case

If ignore case is checked, lines do not need to be cased identically to be considered duplicates. Two identical lines, one in upper case, and one in lower case, would be considered duplicates and removed by this filter. If ignore case is unchecked, the lines must be identical to be considered duplicates. The case checking routines are ANSI aware, so their behaviour may change depending on your locale.

Start column

The comparison can also ignore leading characters if desired by setting the start column higher than 1. This can be used to skip line numbers, which can be used to find duplicates that are not adjacent. To skip line numbers, set the Start Column to 6 (or so), and set the length to 4096, or a length greater than your maximum line length.

Length

The comparison can also ignore trailing characters if desired, by setting the length to less than the length of the line.

See also

Remove duplicate lines
Remove lines found in file
Retain lines found in file


Retain matching lines (grep)/non-matching lines (inverse grep)

 Retain matching/non-matching lines

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

TextPipe will retain any lines that match/don't match the given pattern, all other lines will be deleted. For options, please see the Restrict to matching lines filter. Unlike that filter, this filter is not allowed to have any sub filters.

Note - any non-matching lines are discarded, so you can't really use two Retain Matching Lines filters in a row. Instead, use an or '|' operator in the pattern match.

See also

Special Menu
Remove matching/non-matching lines
Retain lines matching pattern list


Remove lines from list

 Remove lines from list

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

TextPipe will retain all lines that EXACTLY match any of the lines in the given text file, or in the text memo area, all other lines will be deleted (line breaks can be of any format).

See also

Special Menu
Retain lines matching pattern list


Retain lines matching pattern list

 Retain lines matching pattern list

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

TextPipe will retain all lines that match any of the patterns in the given text file (or specified in the text memo area), all other lines will be discarded. Each pattern must be specified on a line of its own (line breaks can be of any format). Each line is interpreted as a perl-style pattern, with options of case-insensitive matching, non-greedy matching, comments not allowed. If you need to change these options, precede each line by the following codes:

Change option command | Result
(?-i) | Case-sensitive matching
(?x) | Comments allowed
(?U) | Greedy matching

See also

Retain matching lines
Retain non-matching lines
Retain lines from list
Retain lines matching pattern list
Remove menu


 Move columns

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

TextPipe will move columns to a new position on the line. The new position is specified assuming that the moved columns have been removed from the line.

Move columns

See the Remove columns filter for how to specify which columns to move.

Move to

The position to insert the moved columns. The new position is specified assuming that the moved columns have been removed from the line.

See also

Special Menu
Move columns
Move fields (CSV, Tab, Pipe etc)


Move fields (CSV, Tab, Pipe etc)

 Move fields (CSV, Tab, Pipe etc)

Submit feedback on this topic

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

TextPipe will move CSV-delimited fields to a new position on the line. The new position is specified assuming that the moved fields have been removed from the line. TextPipe ensures that all the delimiters on the line are correctly maintained, both at the end of the line and where the moved fields are inserted.

Move fields

See the Remove delimited fields (CSV, Tab, Pipe etc) for how to specify which fields to move.

Move to

The position to insert the moved fields. The new position is specified assuming that the moved fields have been removed from the line.

See also

Special Menu
Move columns


 Copy columns

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

TextPipe will copy columns to a new position on the line.

Copy columns

See the Remove columns filter for how to specify which columns to move.

Copy to

The position to insert the copied columns.

See also

Special Menu
Copy columns
Copy fields (CSV, Tab, Pipe etc)


Copy fields (CSV, Tab, Pipe etc)

 Copy fields (CSV, Tab, Pipe etc)

Submit feedback on this topic

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

TextPipe will copy CSV-delimited fields to a new position on the line. TextPipe ensures that all the delimiters on the line are correctly maintained, both at the end of the line and where the copied fields are inserted.

Copy fields

See the Remove delimited fields (CSV, Tab, Pipe etc) for how to specify which fields to copy.

Copy to

The position to insert the copied fields.

See also

Special Menu
Copy columns
Copy fields

Contact Us Support Community Tutorials and User Guides (online)
© 1999-2005 Crystal Software . All rights reserved.


Repeat file's contents

 Repeat file's contents

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

The file's contents are repeated the specified number of times. This filter needs to keep a copy of the entire file in memory so it can output it a given number of times. This filter is useful for generating test data sets of a given size.

See also

Reverse line order


Randomize lines

 Randomize lines

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter put lines into random order. This is useful when a random sample of data is required for statistical purposes - just follow this filter with a head/tail of file filter.

The lines output will differ from one run to the next. The order the lines are chosen is determined by a pseudo-random number generator.


Reverse line order

 Reverse line order

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Each line is output reversed from left to right. This can be useful to

  • extract domain names from web site log files - use this filter to reverse each line, use an extract matches filter of [\w\d]+\.[\w\d]+ to extract each domain name, then reverse each line again.
  • reversing lines of Hebrew text that are in mirror script

See also

Reverse line order


Reverse line order

 Reverse line order

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

The order of the input lines is reversed i.e. the last line comes out first and the first line comes out last.

A file is read entirely into RAM before being reversed, so be wary of reversing files that are larger than your machine's RAM size.

See also

Reverse each line


Restrict filters

 Restriction Filters

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up
Restrict matching lines (Grep)
Restrict non-matching lines (inverse grep)
Restrict each line in turn
Restrict to delimited fields (CSV, Tab, Pipe etc)
Restrict to tag/attribute
Restrict to filename
Restrict line range
Restrict column range
Restrict to byte range
Restrict record

These filters restrict the amount of text that is affected by sub filters i.e. without affecting the main text.

Restrict to line range
Restrict to column range
 Restrict to matching lines (Pro version only)
Restrict to non-matching lines (Pro version only)
Restrict to each line in turn
Restrict to delimited fields (CSV, Tab, Pipe etc)
Restrict to HTML/XML element or attribute
Restrict to filename
Restrict to blocks
Restrict to byte range

Note - sub filters treat each section of restricted text as a complete file.

See also

Lesson 5 - Restrictions and sub filters


Restrict to line range

 Restrict line range

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter restricts sub filters to operate on a range of lines. The lines can be selected by using an offset from the beginning or end of the file. All selected lines can be completely removed by using the Delete All filter as the sub filter. This filter can be used like a Unix head or tail filter.

See also

Sub filters
Restriction filters


Restrict to column range

 Restrict column range

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter restricts sub filters to operate on a range of columns. The columns can be selected by using an offset from the beginning or end of the line. All selected columns can be completely removed by using the Delete All filter as the sub filter. This filter can be made a sub filter of a Restrict line range filter.

If you're working with EBCDIC data, you should not use this filter, because EBCDIC data can contain embedded new line characters in packed or zoned decimal fields. Use the Restrict to Byte range filter instead.

See also

Sub filters
Restriction filters


Restrict to matching lines

 Restrict matching lines (Grep)

Submit feedback on this topic

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

This filter restricts sub filters to operate on lines that match a pattern or regular expression.

Note: This filter allows you to use the regular expression characters, ^ and $.

^ match from the start of a line (or at the start of the file)

$ match at the end of a line (or at the end of the file)

Restrict to Lines Matching

The string or pattern to find. The string can be found anywhere on a line. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering special characters and inserting pattern match characters.

Pattern type

Determines which style of pattern match to use. The default is to use powerful perl-style pattern matching, but you can also use newer and easier EasyPatterns, as well as Word-style and Brief-style patterns. Older egrep-style pattern matching is offered for backward compatibility.

Match Case

When checked, the case of the string found must be identical to the Match Pattern exactly.

Output Count of Matches

When checked, outputs a count of the number of matches in each file instead of the actual matches.

Include Filename

When checked, outputs the filename prior to showing the match information.

Include Line Numbers

When checked, shows the line numbers of each match with the match text.

Context Lines

Context Lines Before the Match

This option outputs context lines prior to the matching line. This can be very useful when you want to see the lines surrounding a matching line. Enter a value larger than 0 to determine the number of context lines to be output. TextPipe may not be able to output the desired number of context lines before the matching line if the matching line is near the start of the file. In this case TextPipe will output as many prior context lines as are available. Note also that if two subsequent lines match then the resulting output may be difficult to understand.

Context Lines After the Match

This option outputs context lines after the matching line. This can be very useful when you want to see the lines surrounding a matching line. Enter a value larger than 0 to determine the number of context lines to be output. TextPipe may not be able to output the desired number of context lines after the matching line if the matching line is near the end of the file. In this case TextPipe will output as many trailing context lines as are available. Note also that if two subsequent lines match then the resulting output may be difficult to understand.

See also

Extract matches
Restrict to non-matching lines
Restriction filters


Restrict to non-matching lines

 Restrict non-matching lines (inverse grep)

Submit feedback on this topic

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

(Pro version only)

This filter restricts sub filters to operate on lines that DO NOT match a pattern or regular expression.

This filter is the exact opposite of the Restrict to matching lines filter. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering special characters and inserting pattern match characters.

See also

Extract matches
Restrict to matching lines
Restriction filters


Restrict to delimited fields (CSV, Tab, Pipe etc)

 Restrict to delimited fields (CSV, Tab, Pipe etc)

Submit feedback on this topic

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter restricts sub filters to operate on a range of fields, delimited by a given character.

Choose the delimiter that separates your fields

You can choose a predefined delimiter character, or select your own. The trailing delimiter (if any) is included in the restricted text, so the entire field can be completely removed by using the Delete All filter as the sub filter (or you can use a Remove delimited fields (CSV, Tab, Pipe etc) filter). When Comma (.csv) is chosen, TextPipe automatically handles single and double quoted strings, with embedded line feeds.

First Row Contains Field Names

If the first line of the file contains Field Names, check this box so that TextPipe can count how many fields are expected. It can then determine if a field has embedded CR/LF characters and spans multiple lines. TextPipe can also determine this without a header if the fields are properly double-quoted - TextPipe will notice the missing double quote and continue reading the record from the following line.

Restrict Fields

Several different means of specifying the fields to restrict are provided:

  1. Start field, end field. This removes all text including and between the specified fields.
  2. Start field, width. Removes Width fields starting from (and including) field Start.
  3. End field, width. Removes Width fields backwards starting from (and including) field End.
  4. Start field to end of line. Removes all fields from the Start field to the very end of the line.
  5. Width to end of line. Removes Width fields backwards starting from (and including) the last field.

Process fields individually

When this field is unchecked, the entire field range is returned as a single string. It is often easier for the sub filters to process each field individually, in which case it is checked, and the next two options apply.

Exclude column delimiter

When checked, this excludes any trailing delimiter from the restricted text, making it easier for sub filters to work with.

Exclude quotes (if present)

When checked, this excludes any surrounding quotes from the restricted text, making it easier for sub filters to work with.

See also

Sub filters
Restriction filters


Restrict XML-delimited field

 Restrict XML-delimited field

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter selects an XML-delimited field for further processing, such as a search/replace inside it, or a case conversion etc.

For example, if your data looked like this, and you wanted to UPPERCASE the CompanyName field:

<TextPipeRow>
<CustomerID>ALFKI</CustomerID>
<CompanyName>Alfreds Futterkiste</CompanyName>
<ContactName>Maria Anders</ContactName>
<ContactTitle>Sales Representative</ContactTitle>
<Address>Obere Str. 57</Address>
<City>Berlin</City>
<Region></Region>
<PostalCode>12209</PostalCode>
<Country>Germany</Country>
<Phone>030-0074321</Phone>
<Fax>030-0076545</Fax>
</TextPipeRow>

Specify the 'Restrict to tag' name to select as 'CompanyName'. Then use a sub filter of UPPERCASE. The resulting output looks like this:

<TextPipeRow>
<CustomerID>ALFKI</CustomerID>
<COMPANYNAME>ALFREDS FUTTERKISTE</COMPANYNAME>
<ContactName>Maria Anders</ContactName>
<ContactTitle>Sales Representative</ContactTitle>
<Address>Obere Str. 57</Address>
<City>Berlin</City>
<Region></Region>
<PostalCode>12209</PostalCode>
<Country>Germany</Country>
<Phone>030-0074321</Phone>
<Fax>030-0076545</Fax>
</TextPipeRow>

Note - if you want to change text inside the XML tag itself, use a Find text inside an HTML tag filter.

See also

Sub filters
 Restrict to delimited fields (CSV, Tab, Pipe etc)


Restrict to filename

 Restrict to filename

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter uses sub filters to operate on the filename. Using this filter, You can rename files, change their folders, UPPERCASE or lowercase the filenames, search and replace parts of the filename and much more!

Note - because sub filters treat each section of restricted text as a complete file, you cannot use a normal line number filter to add numbers to filenames. Instead, use a VBScript-based line number script.

See also

Sub filters
 Restrict to delimited fields
Restriction filters


Restrict to blocks

 Restrict record

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

A Restrict to Blocks filter groups every N bytes together and sends each group to its sub filters. Each group appears to be a "new file" to the sub filters.

For example, a Restrict to Bytes filter specifies a range that only occurs once in each file e.g. Bytes 1 to 3. By placing the Restrict to Bytes filter inside the Restrict to Blocks filter, the Byte range effectively occurs multiple times -

  1. Bytes 1 to 3 of the first group
  2. Bytes 1 to 3 of the second group
  3. Bytes 1 to 3 of the third group
  4. and so on.

If you have a known record length, use this filter to ensure each record is processed as a single unit.

When combined with a Secondary Output Filter as a sub filter (with Append Mode on), records can be selectively written to a new file.

See also

Sub filters
Restriction filters


Restrict to byte range

 Restrict to byte range

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter restricts sub filters to operate on a range of bytes. The bytes can be selected by using an offset from the beginning or end of the containing subfilter, usually a Restrict to blocks filter. Selected bytes can be completely removed by using the Delete All filter as the sub filter.

This filter is handy for limiting the data sent to a packed decimal or zoned decimal expansion filter.

See also

Sub filters
Restriction filters


Map Filters

 Map Filters

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up
New map
Unicode maps
Available maps
Refresh map list
Base map on filter list

Found under the "Maps" menu, these filters map each character of input to an output character or string.

A map defines a mapping for every single input character (all 256 of them). Each ASCII code is represented by a row in a grid – the left column gives the ASCII code in decimal ("Dec"), the next column gives the ASCII code in Hex ("Hex"), the next column shows the ASCII representation of the character ("ASCII"), and the right hand column shows the string that it will be mapped to ("Replacement text"). Note that if the system font does not provide a definition for a particular ASCII character (as is usually the case for ASCII codes 0 to 31 and 127, but there may be more), then it is represented as a thick vertical bar. Cells that are output differently to the input are shown with a yellow background - this makes it easy to scan through the list looking for changes.

When the replacement text cannot be normally displayed, characters are displayed using an escape code e.g. \r\n is the code for a DOS end of line. Any combination of characters can be entered in this way. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy and entering special characters.

If all the replacements are a single byte, then TextPipe will used an optimized filter which is 3 times faster than the custom case (which handles replacements of zero characters, one character or more).

Reset

A cell is Reset when it passes through unchanged i.e. the output character is the same as the input character.

Reset All

All cells can be reset by clicking "Clear All".

Clear

A cell is cleared or empty when nothing is output when the character is encountered in the input stream.

Clear All

All cells can be cleared by clicking "Clear All".

Reverse

If the character mapping is 1:1 (i.e. each input character gets mapped to a single output character), the "Reverse" button can be clicked to form the Reverse mapping translation.

Scroll to

Clicking in the Scroll To field and pressing a keyboard character will cause the grid to immediately scroll to show the mapping for that character. This makes it easy to find characters when you're not sure where in the list they appear.

Open and Save

Maps can be saved to and opened from .MAP files by clicking on the Save and Open icons displayed when a map filter is selected for editing.

See also

Refresh map list
New map
Base map on filter list
Available maps
 Unicode maps


New map

 New map

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Adds a new map filter, using values from the last edited map as defaults.

See also

Refresh map list
Base map on filter list
Available maps
 Unicode maps


Unicode maps

 Unicode maps

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up
Multi-maps

A Unicode map allows 2 byte (UTF-16) or 4 byte (UTF-32) Unicode characters to be mapped to a sequence of zero or more output characters in the same way that single byte characters are mapped. The primary difference between ANSI maps and Unicode maps is that a Unicode map only specifies characters that are changed - non-specified characters are controlled by a separate setting (otherwise you would have an unwieldy list of 65536 to 1 million entries to edit and work with).

The Unicode map expects characters to be found in big endian order, i.e. MSB first, so the Unicode code point x00FA would be found in the file as two consecutive characters, 0x00 0xFA, NOT 0xFA 0x00. If the Unicode map does not work for you, you can check your input file format using the Convert\Hex Dump filter, and if necessary, use the Swap UTF-16 word order or Swap UTF-32 word order filter.

A Unicode map filter looks like this:

Type a value in the output string column to define how a Unicode character gets remapped. Click the up and down buttons in the Start Range (Hex) or End Range (Hex) columns or type a new number to change the range of Unicode characters being remapped. New values can be added on the last row, or using the Populate Values panel.

Delete Selected Rows

Deletes the currently selected rows from the grid. Character ranges not defined in the grid are controlled by the Non-entered characters panel.

Open Map File

Opens a Unicode map file, replacing the existing map grid (not the non-entered characters panel values). Maps can be loaded from Excel worksheets (.XLS), Comma Separated Value files (.CSV, the default when the file extension is not recognized) and Tab delimited value files (.TAB).

Save Map to File

Saves the current Unicode map grid to a file (not the non-entered characters panel values). Maps can be saved to Excel worksheets (.XLS), .CSV Comma Separated Value file (the default) or a .TAB Tab-delimited value file (only when the file extension is .TAB). Values are saved exactly as they are shown on screen.

UTF Mode

This drop down specifies whether each input character is 2 bytes (UTF-16) or 4 bytes (UTF-32).

Non-entered characters

This panel controls what happens to characters not found in any range in the grid.

  • Pass through unchanged - characters not found in the grid are passed through unchanged
  • Pass to sub filter. Typically the sub filter is a Script filter - see below for a sample script.
  • Replace with value - characters not found in the grid are replaced with the set value. Using %x (or %4.4x for better formatting) in the replacement value as a placeholder for the hex value of the character. Using %d in the replacement value is a placeholder for the decimal value of the character. e.g.

    Invalid character found 0x%4.4x
  • Remove - characters not found in the grid are removed

Populate values

The populate values group makes it easy to set large groups of values. First enter in a starting value and an ending value (in decimal), then click the button corresponding to your choice:

[Clear] Add the specified characters to the grid and set their mapping so that nothing is output when they are encountered.

[Default] Add the specified characters to the grid, and set their mapping so that they are passed through unchanged.

Sample sub filter VBScript

'Log unknown Unicode values
to TextPipe error log
dim a
dim errorCount
const maxErrors = 20

function processLine(line, EOL)
if len(line & EOL) = 2 then
'line & EOL is the two characters of the Unicode code point
errorCount = errorCount + 1
if errorCount < maxErrors then
a = Asc(mid(line & EOL,1,1)) * 256 + Asc(mid(line & EOL,2,1))
TextPipe.logError "Invalid code point 0x" & hex( a )
'msgBox "Invalid code point 0x" & hex( a )
elseif errorCount = maxErrors then
TextPipe.logError "Maximum code point Errors reached: " & errorCount
end if
end if

'return nothing - invalid code point is absorbed
processLine = ""
end function

sub startJob()
errorCount = 0
end sub

sub endJob()
end sub

function startFile()
startFile = ""
end function

function endFile()
endFile = ""
end function

See also

Refresh map list
New map
Base map on filter list
Multi-maps
Available maps
Swap UTF-16 word order
Swap UTF-32 word order
Make Big Endian
Make Little Endian
Unicode conversion


Base map on filter list

 Base map on filter list

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Adds a new map based on the filters currently in the filter list. This is very useful if you need to customise any of the existing filters eg the EBCDIC -> ASCII filter. By basing a new map on the current filter list, you can effectively "copy" the existing filter, and then tweak it as desired.

See also

Refresh map list
New map
Available maps
 Unicode maps


Refresh map list

 Refresh map list

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

If you copy or move map files to TextPipe’s folder, use this item or press F5 to force TextPipe to display it.

See also

New map
Base map on filter list
Available maps
 Unicode maps


Available maps

 Available maps

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Any maps that have been defined by the user (and are saved in the same folder as TextPipe.exe) will be listed in this menu.

To add a saved map to the filter list, just select it from this menu.

See also

Refresh map list
New map
Base map on filter list
Unicode maps


Tools Menu

 Tools Menu

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up
Command line wizard
Customize shell extensions
Show command line

Prepare filter for distribution

This item clears the Trial Input and Output areas and the Scratch Pad area, so any data that you have been testing with won't be seen by the people who receive your filter.

Disable prompting

This item checks the filter for any form of user prompting and offers to disable it. This is very useful when you need to automate a complex filter, because it saves having to check each and every search/replace filter to see if prompting is enabled, and also checking the input filter to see if Prompt on Binary Files or Prompt Before Each File is checked.

Put sample in Trial Input area

Depending on the Input Filter's selection, either grabs a sample of text from the clipboard, or grabs the entire first file in the file list. An error message is shown if there are no files in the file list or the file cannot be found.

Move Trial Output to Trial Input

Copies the text in the Trial Output area back to the Trial Input area.

Verify filter list

This command checks all the filters in the filter list for errors. This action is always performed before a filter list is run.

Filter wizard

The Filter Wizard provides a step-by-step method for beginners to construct a TextPipe filter list.

Command line wizard

Gives examples of invoking the current filter from the command line.

Customize shell extensions

This option lets you customize the items that appear in the right-click menu in Windows Explorer when you click on a file, folder or drive.

Show command line

Displays the command line used to start TextPipe. This can be used to trouble-shoot command line parameters if you use macros to substitute variables.

Clipboard viewer

Displays the Windows clipboard viewer, which displays the current contents of the Windows clipboard. This can be very handy for testing a set of filters, because you can easily see what happens to the text, both before and after.

If you didn’t install this component of Windows, you can do so by checking it under Accessories

Notepad

Displays NotePad, a small editor useful for editing plain text files (files that contain no formatting like bold, fonts etc). This can be very handy for generating test data for a set of filters. NotePad is automatically installed with Windows.

If you didn’t install this component of Windows, you can do so by checking it under Accessories

Wordpad

Displays WordPad, a small editor useful for editing both plain text files (files that contain no formatting like bold, fonts etc) and Word documents, Rich Text Format (RTF) documents and Windows Write (WRI) documents. This can be very handy for generating test data for a set of filters.

If you didn’t install this component of Windows, you can do so by checking it under Accessories

Character map

Displays Windows Character Map. This program is usually used to insert special characters into documents, and it can also be used to determine the ASCII code of a character.

If you didn’t install this component of Windows, you can do so by checking it under Accessories

Calculator

Displays the Windows Calculator.

If you didn’t install this component of Windows, you can do so by checking it under Accessories

Show log file

Moved to Filter options window.

Truncate log file

Moved to Filter options window.


Command line wizard

 Command line wizard

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This function is found in the Tools menu\Command line wizard.

The command line wizard displays actual examples of how to run the current filter from the command line in various modes. Click the Refresh button to display new examples if the current filter changes.

See also

Command line parameters


Customize shell extensions

 Customize shell extensions

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This option lets you customize the TextPipe menu items that appear in the right-click menu in Windows Explorer when you click on a file, folder or drive.

This function is found in the Tool menu\Customize shell extensions.

See also

Command line parameters

Menu name

The name given to the filter in the Windows Explorer right-click context menu.

Comment

The comment is displayed on the Windows Explorer's status line at the base of the window. It gives more information about what a filter does.

Filter file name

The name of the filter file to use. This field is optional - if you do not need to use a filter file you don't need to specify one.

Command

The TextPipe command line to use. This allows you to tailor how TextPipe is run - whether it runs and then exits immediately (using a '/g /q' on the end), etc.

%f is replaced by the filter file name (if one is specified).

%p is replaced by the list of selected files, each one is double quoted automatically

Add new rows

Move the cursor to the very last cell, then press Tab. This creates a new row below the last row.

Delete a row

Select the row by clicking it, then click the Delete Row button.

Change the order of items in the menu

Use the row selector on the left of each row to drag the row to a new position

Ok

Saves the changes and closes the window.

Apply

Saves the changes but does not close the window, allowing you to preview the changes in Explorer.

Cancel

Does not save any changes and closes the window.


Show command line

 Show command line

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This function is found in the Tools menu\Show command line.

This function is useful if you are not sure how your command line arguments are being passed to TextPipe. TextPipe does not show quoted parameters in the grid - the actual grid contents show how TextPipe has interpreted the command line. The string adds quotes back in for parameters that contain spaces - it is not an exact representation of the original command line. The text also shows the current path (PATH), and all the environment variables passed to TextPipe when it was started.

See also

Command line parameters


Window Menu

 Window Menu

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Cascade

Arranges each window so that they overlap.

Maximize

Increases the size of the active window so that it fills the TextPipe window area entirely.

Minimize

Shrinks the current window to just the height of its title bar.

Minimize all

Shrinks all windows to just the height of their title bar. To restore a minimized window: If the window or another filter is showing, click the name of the document you want to restore on the Window menu.

Restore

Grows a minimized window back to its pre-minimized size, or shrinks a maximized window back to its pre-maximized size.

Arrange Icons

Arranges the title bars of the minimized windows.

Next

Changes focus to the next child window.

Previous

Changes focus to the previous child window.


Help Menu

 Help Menu

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up
Check for updates

Contents

Displays the contents page of the TextPipe online help.

Index

Displays the index page of the TextPipe online help.

Tutorial

Displays the beginners tutorial and introduction to TextPipe.

Show Tip of the Day

Shows a series of handy hints for using TextPipe.

Related products

Displays a list of applications and products related to TextPipe at the Crystal Software website.

Buy TextPipe Now!

Opens your web browser at http://www.crystalsoftware.com.au/buy.html to purchase TextPipe.

Order form

Opens a text-based order form to order TextPipe.

TextPipe Home Page

Opens your web browser at http://www.crystalsoftware.com.au/textpipe.html

Crystal Software Home Page

Opens your web browser at http://www.crystalsoftware.com.au

Check web for TextPipe updates

Opens the Check for Updates dialog.

Subscribe to TextPipe update list

Join the TextPipe email list to receive notification of program updates and new releases. To join, send an email to

textpipe-subscribe@yahoogroups.com

or go the following web page.

Subscribe to TextPipe discussion list

Join the TextPipe email list to talk to other TextPipe users and gain knowledge on how to use TextPipe better. To join, send an email to

textpipe-discuss-subscribe@yahoogroups.com

or go the following web page.

Technical support

Opens your web browser at http://www.crystalsoftware.com.au/support.html

Suggestion box

Opens your web browser at a form you can fill in to send feedback to Crystal Software.

About

Displays TextPipe's About window.


Check for updates

Found in the Help menu, this option lets you check if you have the most up-to-date version. This dialog box is automatically displayed 5 days after the software is first installed.

Unless the Don't ask me again until date is set in the future, the software will check the web for updates every time it is started. If Windows is setup to automatically dial-out if no Internet connection is detected, then checking for updates may cause the Dial-Up dialog to appear.

Check web for updates now

Selecting this radio button and then clicking on Ok results in an access to the Crystal Software web site to determine if your software has been superceded. If your copy is out of date, the software will confirm that you wish to download a new version. If so, it will open a web browser at a download page. If you do not wish to download, it will check again in 30 days time.

Don't ask me again until

Selecting this radio button and then setting a date in the calendar drop-down allows you to set a date after which it will again attempt to check for an update. Set this date well in the future if you don't want it to check for updates.

Initially this date is set to 5 days after the software is installed.


An Introduction to Pattern Matching

 An Introduction to Pattern Matching

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up
Regular Expression Examples

  • For more practical examples, take a look at the section, Examples Using Regular Expressions.
  • Perl/PCRE pattern matching reference
  • For further reference, we recommend Mastering Regular Expressions: Powerful Techniques for Perl and Other Tools (O'Reilly Nutshell) -- Jeffrey E. Friedl (Editor), Andy Oram (Editor); Paperback

Why do we need patterns?

We need patterns to describe text that is inexact, such as 'all words starting with s', 'all words starting with t and ending with m', or 'all 9 digit numbers'. We might be able to list all the possible variations, but it would be impractical and annoying.

We use a pattern matching language to describe a pattern. A pattern matching language uses symbols to describe both normal characters (that are matched exactly) and meta characters (that describe special operations, such as alternatives and repeating sections). Patterns are almost always described using what is called a regular expression, so called because the pattern matching language fulfills some mathematical properties that we don't need to worry about. Patterns can become very complicated very quickly, so let's start with some simple examples.

We've all done a search and replace where we don't care about the case of the search string (Case Sensitive is Off). For example, if I search for 'Car', I also want to search for 'car, 'CAR' and any other case variations. We are implicitly searching for an upper or lower case 'C', followed by an upper or lower case 'A' followed by an upper or lower case 'R'. This is an example of a simple pattern.

In a pattern matching language, we could express this as follows:

[Cc][Aa][Rr]

Each set of square brackets is a character class, meaning 'find one of any of these characters' - a 'C' or 'c', then a 'A' or 'a', then a 'R' or 'r'. If we changed this pattern to

[BbCc][Aa][Rr]

then we also could find 'Bar', 'bar' and 'BAR' (and others). Of course, if we just toggled the Case Sensitivity flag then it wouldn't matter if we used

[bc][a][r] or [BC]AR or [bC]Ar or even [bc]ar

Note that the letters outside of the square brackets simply represent themselves - they are not special.

Here are some more character classes:

[aeiou] | matches any lower case vowel
[^aeiou] | matches any character that is NOT a lower case vowel
[0123456789] | matches any digit
[^0123456789] | matches any character that is NOT a digit
] NOT special, NOT the end of a character class when on its own
[]abc]
[abc\]]
[^]abc]
 matches a, b, c and ]
matches a, b, c and ]
matches everything except a, b, c and ]

If a closing square bracket is required as a member of the class, it should be the first data character in the class (after an initial circumflex, if present) or escaped with a backslash. It is not possible to have the literal character "]" as the end character of a range.

[0123456789^]
[\^0123456789] 		Matches 0 to 9 and ^
Matches 0 to 9 and ^

If a circumflex is actually required as a member of the class, ensure it is not the first character, or escape it with a backslash.

[0-9]
[d-m]
[0-9a-zA-Z]
[-0-9]
[0-9-]
[\000-\037]

Matches 0 to 9
Matches d to m
Matches 0 to 9, a to z, A to Z
Matches 0 to 9 and -
Matches 0 to 9 and -
Matches ASCII characters 0 to 31 (non printable characters)

The minus (hyphen) character can be used to specify a range of characters in a character class. For example, [d-m] matches any letter between d and m, inclusive. If a minus character is required in a class, it must be escaped with a backslash or appear in a position where it cannot be interpreted as indicating a range, typically as the first or last character in the class. Ranges operate in ASCII collating sequence. They can also be used for characters specified numerically, for example [\000-\037].

TextPipe also provides some convenient short cuts for commonly used classes. They may be used either on their own or inside a character class.

. | Any character, including newline (by default)
\d | any decimal digit
\D | any character that is not a decimal digit
\s | any white space character. space, formfeed, newline, carriage return, horizontal tab, and vertical tab
\S | any character that is not a white space character
\w | any "word" character. A "word" character is any letter or digit or the underscore character. The definition of letters and digits may vary, for example, in the "fr" (French) locale, some character codes greater than 128 are used for accented letters, and these are matched by \w.
\W | any "non-word" character

You can also describe 'special' or unprintable characters using the following patterns:

\t | tab (decimal 9, hex 09)
\f | form feed (decimal 12, hex 0C)
\n | new line (decimal 10, hex 0A)
\r | carriage return (decimal 13, hex 0D)
\a | alarm, that is, the BEL character (decimal 7, hex 07)
\e | escape (decimal 27, hex 1B)
\cx | "control-x", where x is any character The precise effect of "\cx" is as follows: if "x" is a lower case letter, it is converted to upper case. Then bit 6 of the character (hex 40) is inverted. Thus "\cz" becomes hex 1A, but "\c{" becomes hex 3B, while "\c;" becomes hex 7B.
\xhh | character with hex code hh After "\x", up to two hexadecimal digits are read (letters can be in upper or lower case).
\ddd | character with octal code ddd, or back reference After "\0" up to two further octal digits are read. In both cases, if there are fewer than two digits, just those that are present are used. Thus the sequence "\0\x\07" specifies two binary zeros followed by a BEL character. Make sure you supply two digits after the initial zero if the character that follows is itself an octal digit.

Repetition - quantifiers

Often you are looking for a pattern that is repeated. TextPipe provides a number of different methods to specify how many times a pattern is allowed to repeat. Each method is called a quantifier.

The general repetition quantifier specifies a minimum and maximum number of permitted matches, by giving the two numbers in curly brackets (braces), separated by a comma. The numbers must be less than 65536, and the first must be less than or equal to the second. For example:

z{2,4}

matches "zz", "zzz", or "zzzz". A closing brace on its own is not a special character. If the second number is omitted, but the comma is present, there is no upper limit; if the second number and the comma are both omitted, the quantifier specifies an exact number of required matches. Thus

[aeiou]{3,}

matches at least 3 successive vowels, but may match many more, while

\d{8}

matches exactly 8 digits. An opening curly bracket that appears in a position where a quantifier is not allowed, or one that does not match the syntax of a quantifier, is taken as a literal character. For example, {,6} is not a quantifier, but a literal string of four characters. For convenience (and historical compatibility) the three most common quantifiers have single-character abbreviations:

* is equivalent to {0,}
+ is equivalent to {1,}
? is equivalent to {0,1}

Normally TextPipe will try to repeat matches as many times as possible - this is known as greedy (or maximal) matching. Using the Search and Replace Pattern Options dialog you can change this to non-greedy (or minimal) matching. If a quantifier is followed by a question mark, it inverts the default 'greediness' of the quantifier.

Pattern | Subject text (match shown in bold) | Greedy/Non greedy
<.*> | <html><p>Hello there</p></html> | Non-greedy, minimal match
<.*> | <html><p>Hello there</p></html> | Greedy, maximal match (matches entire text)
<.*?> | <html><p>Hello there</p></html> | Non-greedy, minimal match
<.*?> | <html><p>Hello there</p></html> | Greedy, maximal match (matches entire text)

Using these quantifiers we can now build the following examples:

a+ | a, aa, aaa, aaaa etc
\d+ | any decimal integer
\w+ | any single word
\d{2,3} \d{3,4} \d{4} | A 7 or 8 digit phone number with a 2 or 3 digit area code
s.t | Matches the real words set, sat, sit (and others)

Alternatives

Vertical bar ('pipe') characters are used to separate alternative patterns. For example, the pattern

gilbert|sullivan

matches either "gilbert" or "sullivan". Any number of alternatives may appear, and an empty alternative is permitted (matching the empty string). The matching process tries each alternative in turn, from left to right, and the first one that succeeds is used. If the alternatives are within a sub pattern (defined below), "succeeds" means matching the rest of the main pattern as well as the alternative in the sub pattern.

Positional matching or anchors

Sometime you only want a pattern to match when it is found at the start of a file, or the end of a line, or when it's a whole word. Various positional operators force the match to only occur when it's found in a specific position.

^ Almost always used at the start of a pattern, the circumflex forces the match to only occur at the start of a line (i.e. after a \n or ASCII 10) or at the start of the file.

Circumflex need not be the first character of the pattern if a number of alternatives are involved, but it should be the first thing in each alternative in which it appears if the pattern is ever to match that branch.

Instead of Circumflex, consider using the following assertion:

(?<=\r|\n|\A)

The reason for this change is a little technical. TextPipe's pattern matching engine expects Unix end of line characters (\n or ASCII 10), so Circumflex will work for Unix or PC files, but fails utterly on Macintosh files unless you use the assertion above or explicitly convert the file first.

$

Almost always used at the end of a pattern, the dollar character forces the match to only occur at the end of a line (i.e. before a \n or ASCII 10), or at the end of file.

Dollar need not be the last character of the pattern if a number of alternatives are involved, but it should be the last item in any branch in which it appears. Dollar has no special meaning in a character class.

Instead of Dollar, consider using the following assertion:

(?=\r|\n|\z)

The reason for this change is a little technical. TextPipe's pattern matching engine expects Unix end of line characters (\n or ASCII 10), so Dollar will work on Unix files, fail utterly on Macintosh files, and on PC files it will include the preceding \r (ASCII 13) as part of the line unless you use the assertion above or explicitly convert the file first.

\b

word boundary

A word boundary is a position in the subject string where the current character and the previous character do not both match \w or \W (i.e. one matches \w and the other matches \W), or the start or end of the string if the first or last character matches \w, respectively.

\B

not a word boundary

\A

start of file

\Z

end of file or new line at end of file

\z

end of file

The difference between \Z and \z is that \Z matches before a new line that is the last character of the string as well as at the end of the string, whereas \z matches only at the end.

Sub Patterns and the Replacement String

Sub patterns are delimited by parentheses (round brackets), which can be nested. Marking part of a pattern as a sub pattern does two things:

1. It localizes a set of alternatives. For example, the pattern

cat(aract|erpillar|)

matches one of the words "cat", "cataract", or "caterpillar". Without the parentheses, it would match "cataract", "erpillar" or the empty string.

2. It sets up the sub pattern as a capturing sub pattern. The text matches by a capturing sub pattern can be referred to later (such as in the replacement string) using the macros $0 (for the full matching text), $1 (for the first sub pattern), $2 ... $9, $a ...$z etc. Opening parentheses are counted from left to right (starting from 1) to obtain the numbers of the capturing sub patterns.

For example, if the string "the red king" is matched against the pattern

the ((red|white) (king|queen))

the captured substrings are "red king", "red", and "king", and are numbered 1, 2, and 3.

The fact that plain parentheses fulfill two functions is not always helpful. There are often times when a grouping sub pattern is required without a capturing requirement. If an opening parenthesis is followed by "?:", the sub pattern does not do any capturing, and is not counted when computing the number of any subsequent capturing sub patterns. For example, if the string "the white queen" is matched against the pattern

the ((?:red|white) (king|queen))

the captured sub strings are "white queen" and "queen", and are numbered 1 and 2. The maximum number of captured sub strings is 36 (0-9, a-z), and the maximum number of all sub patterns, both capturing and non-capturing, is 200.

Using parentheses appropriately in the search expression lets the program remember found text, to be used as replacement text. The simplest example of this, without the need for parentheses, is the complete found string, represented by the ‘$0’ character in the replacement string. For example, if your search Regular Expression was ‘test|trial|experiment’, and your replacement string was ‘<b>$0</b>’, every instance of the word ‘test’ in your document would be replaced by ‘<b>test</b>’, and similarly for ‘trial’ and ‘experiment’ (assuming this was an HTML document, this would have the effect of bolding these words). Note that if you want to include an actual ‘$’ in your replacement text, escape it, as in, ‘$$’.

You can use parentheses to remember specific parts of the found text. For example, the Regular Expression ‘<img src="([^"]+)">’ would find and match image tags in an HTML document (assuming they were formed exactly like this), and would remember the source image file. You can recall this by using ‘$1’ in your replacement string. So if your replacement string was ‘-- Image File: $1 --’, and the HTML file processed contained the string ‘<img src="/images/test.gif">’, that string would be replaced by ‘-- Image File: /images/test.gif --’.

You can remember multiple parts of the found text. For example, the Regular Expression ‘<img src="([^"]+)" alt="([^"]+)">’ would find and match a string such as ‘<img src="/images/test.gif" alt="My Cool Image">’. If the replacement string was ‘$2 ($1)’, then this image would be replaced by ‘My Cool Image (/images/test.gif)’.

Performance

The time which a search for a Regular Expressions takes can range from unimportant to unbearably slow. Seemingly small changes in a Regular Expression can make a world of difference. For example, the regular expression

‘[^"]*("[^"]+")[^"]*’

finds and matches quoted strings (which it remembers), as well as surrounding text without quotes. This particular Regular Expression executes quickly enough (counted in seconds, or less) if the file being processed actually has quoted strings in it. If, however, the file is of a reasonable size (say, 50 KB), and does not have any quoted strings in it, the search for this regular expression will take an incredible amount of time to complete – maybe minutes! It turns out that it is the first piece of the expression that is causing the problem. Removing it makes the search on the 50 KB file nearly instantaneous. Why? Because now the search is smart enough to realize that before it bothers matching anything else, it must at least find a double-quote. When it doesn’t find one, it’s finished it’s search.

So one trick to making fast regular expressions, is to form the regular expression in such a way that it can fail as early as possible. Try to put strings that must be matched, right at the beginning of the regular expression. Apart from this, experiment and see what works well and what doesn’t.

Things To Note - Escaping Meta characters

If you want to find any of the meta characters on their own, you must escape them with a backslash to prevent them being interpreted. When in doubt, quote all non-alphanumeric characters. The meta characters are:

Meta character | Purpose
\ | first character of assertions, such as \w or \d
| | OR; allows matching options
. | match any character
( ... ) | grouping operator; builds $1, $2, etc.
[ ... ] | match any character within brackets
+ | quantifier: match one or more times
? | quantifier: at most one match
* | quantifier: match zero or more times
{n,m} | quantifier: match between n and m times
^ | match at start of line
$ | match at end of line

Further reading

For more information on performance, assertions, back references, limitations, POSIX character classes and more details on the above text, please see the Perl/PCRE pattern matching reference.


Regular Expression Examples

 Regular Expression Examples

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Some Basic Regular Expression Search Examples

What to Match | Operator | Examples
Any single character | . | g.t matches get, got, gut, together, g-t
Any string of characters (one or more) | + | w+e matches were, wwezaa, wwwe.write (longest possible match is used)
Any string of characters (or none) | * | w*e matches were, were, wwezaa, wwwe.write (longest possible match is used)
One of the specified characters | [] | g[eo]t matches get and got but not gut
One of the characters in a range | [-] | [b-p]at matches bat, cat, fat, hat, mat but not rat or sat
One expression or another | (|) | W(in|indows) matches Win or Windows
One or more expressions | ()+ | (at)+ matches atat in catatonic and at in battle
All characters (perhaps on different lines) | .* | h.*d matches helped, Hello World, & Hello (cr lf) Win95 World.
Two strings “nearby” (perhaps on different lines) | .* | the[\0-ÿ]*first matches the first and the (cr lf) very first
One of the characters not in a range | [^-] | [^b-p]at matches at in rat & sat and nothing in bat, cat, fat, hat.
Find all words of length x |  | [^\w]\w\w\w+ will find all 3 letter words, [^\w]a\w\w\w\w finds all 5 letter words starting with 'a'.

Additional Regular Expression Search Examples

Search String | Match
L. | line, list, lane, LogVar1 (when case sensitive is off)
[on][on] | non, no, on
\[ [
[a-z]+ abcdefghijklomopqrstuvwxyz
Note: If used with Case Sensitive off, it will also match ABCDEFGHIJKLMNOPQRSTUVWXYZ
[a-zA-ZÀ-ÿ] | Matches the same as the +[a-z] example above (regardless of case sensitive flag) and also all other special characters for Western European languages.
(one|two|three) | one, two, three
(one|two|three)+ | one, onetwo, twothree
Windows[\0-ÿ]*95 | Will match up to 4096 characters (on several lines) between Windows and 95.
All .* replaces | all the replaces (the .* character matches all chars)
/\*[\0-ÿ]*\*/ | Matches C style comments on several lines if necessary (up to 4096 characters)
01*[0-9].htm | 01346.htm, 01.htm, 016965.htm
[ab]*c | c, abc, bac, abbc, bbac
.include (<|\[)+[a-z0-9_].h*(p)+[\]>] Matches:
#include [stdafx.h]
#include <dos.h>
#include [my_include.hpp]
#include [sr32.h]

See also

Regular expressions


Perl Regular Expressions

 Perl Regular Expressions

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

TextPipe's Perl compatible regular expressions are provided by the PCRE library, described below in detail.

This HTML document has been generated automatically from the original man page. If there is any nonsense in it, please consult the man page, in case the conversion went wrong.

  • PCRE REGULAR EXPRESSION DETAILS
  • BACKSLASH
  • CIRCUMFLEX AND DOLLAR
  • FULL STOP (PERIOD, DOT)
  • MATCHING A SINGLE BYTE
  • SQUARE BRACKETS
  • POSIX CHARACTER CLASSES
  • VERTICAL BAR
  • INTERNAL OPTION SETTING
  • SUBPATTERNS
  • NAMED SUBPATTERNS
  • REPETITION
  • ATOMIC GROUPING AND POSSESSIVE QUANTIFIERS
  • BACK REFERENCES
  • ASSERTIONS
  • CONDITIONAL SUBPATTERNS
  • COMMENTS
  • RECURSIVE PATTERNS
  • SUBPATTERNS AS SUBROUTINES
  • CALLOUTS
  • UTF-8 Support
  • Differences from Perl

PCRE REGULAR EXPRESSION DETAILS

The syntax and semantics of the regular expressions supported by PCRE are described below. Regular expressions are also described in the Perl documentation and in a number of other books, some of which have copious examples. Jeffrey Friedl's "Mastering Regular Expressions", published by O'Reilly, covers them in great detail. The description here is intended as reference documentation.

The basic operation of PCRE is on strings of bytes. However, there is also support for UTF-8 character strings. To use this support you must build PCRE to include UTF-8 support, and then call pcre_compile() with the PCRE_UTF8 option. How this affects the pattern matching is mentioned in several places below. There is also a summary of UTF-8 features in the section on UTF-8 support in the main pcre page.

A regular expression is a pattern that is matched against a subject string from left to right. Most characters stand for themselves in a pattern, and match the corresponding characters in the subject. As a trivial example, the pattern

  The quick brown fox

matches a portion of a subject string that is identical to itself. The power of regular expressions comes from the ability to include alternatives and repetitions in the pattern. These are encoded in the pattern by the use of meta-characters, which do not stand for themselves but instead are interpreted in some special way.

There are two different sets of meta-characters: those that are recognized anywhere in the pattern except within square brackets, and those that are recognized in square brackets. Outside square brackets, the meta-characters are as follows:

  \      general escape character with several uses
  ^      assert start of string (or line, in multiline mode)
  $      assert end of string (or line, in multiline mode)
  .      match any character except newline (by default)
  [      start character class definition
  |      start of alternative branch
  (      start subpattern
  )      end subpattern
  ?      extends the meaning of (
         also 0 or 1 quantifier
         also quantifier minimizer
  *      0 or more quantifier
  +      1 or more quantifier
         also "possessive quantifier"
  {      start min/max quantifier

Part of a pattern that is in square brackets is called a "character class". In a character class the only meta-characters are:

  \      general escape character
  ^      negate the class, but only if the first character
  -      indicates character range
  [      POSIX character class (only if followed by POSIX
           syntax)
  ]      terminates the character class

The following sections describe the use of each of the meta-characters.

BACKSLASH

The backslash character has several uses. Firstly, if it is followed by a non-alphameric character, it takes away any special meaning that character may have. This use of backslash as an escape character applies both inside and outside character classes.

For example, if you want to match a * character, you write \* in the pattern. This escaping action applies whether or not the following character would otherwise be interpreted as a meta-character, so it is always safe to precede a non-alphameric with backslash to specify that it stands for itself. In particular, if you want to match a backslash, you write \\.

If a pattern is compiled with the PCRE_EXTENDED option, whitespace in the pattern (other than in a character class) and characters between a # outside a character class and the next newline character are ignored. An escaping backslash can be used to include a whitespace or # character as part of the pattern.

If you want to remove the special meaning from a sequence of characters, you can do so by putting them between \Q and \E. This is different from Perl in that $ and @ are handled as literals in \Q...\E sequences in PCRE, whereas in Perl, $ and @ cause variable interpolation. Note the following examples:

  Pattern            PCRE matches   Perl matches

  \Qabc$xyz\E        abc$xyz        abc followed by the
                                      contents of $xyz
  \Qabc\$xyz\E       abc\$xyz       abc\$xyz
  \Qabc\E\$\Qxyz\E   abc$xyz        abc$xyz

The \Q...\E sequence is recognized both inside and outside character classes.

A second use of backslash provides a way of encoding non-printing characters in patterns in a visible manner. There is no restriction on the appearance of non-printing characters, apart from the binary zero that terminates a pattern, but when a pattern is being prepared by text editing, it is usually easier to use one of the following escape sequences than the binary character it represents:

  \a        alarm, that is, the BEL character (hex 07)
  \cx       "control-x", where x is any character
  \e        escape (hex 1B)
  \f        formfeed (hex 0C)
  \n        newline (hex 0A)
  \r        carriage return (hex 0D)
  \t        tab (hex 09)
  \ddd      character with octal code ddd, or backreference
  \xhh      character with hex code hh
  \x{hhh..} character with hex code hhh... (UTF-8 mode only)

The precise effect of \cx is as follows: if x is a lower case letter, it is converted to upper case. Then bit 6 of the character (hex 40) is inverted. Thus \cz becomes hex 1A, but \c{ becomes hex 3B, while \c; becomes hex 7B.

After \x, from zero to two hexadecimal digits are read (letters can be in upper or lower case). In UTF-8 mode, any number of hexadecimal digits may appear between \x{ and }, but the value of the character code must be less than 2**31 (that is, the maximum hexadecimal value is 7FFFFFFF). If characters other than hexadecimal digits appear between \x{ and }, or if there is no terminating }, this form of escape is not recognized. Instead, the initial \x will be interpreted as a basic hexadecimal escape, with no following digits, giving a byte whose value is zero.

Characters whose value is less than 256 can be defined by either of the two syntaxes for \x when PCRE is in UTF-8 mode. There is no difference in the way they are handled. For example, \xdc is exactly the same as \x{dc}.

After \0 up to two further octal digits are read. In both cases, if there are fewer than two digits, just those that are present are used. Thus the sequence \0\x\07 specifies two binary zeros followed by a BEL character (code value 7). Make sure you supply two digits after the initial zero if the character that follows is itself an octal digit.

The handling of a backslash followed by a digit other than 0 is complicated. Outside a character class, PCRE reads it and any following digits as a decimal number. If the number is less than 10, or if there have been at least that many previous capturing left parentheses in the expression, the entire sequence is taken as a back reference. A description of how this works is given later, following the discussion of parenthesized subpatterns.

Inside a character class, or if the decimal number is greater than 9 and there have not been that many capturing subpatterns, PCRE re-reads up to three octal digits following the backslash, and generates a single byte from the least significant 8 bits of the value. Any subsequent digits stand for themselves. For example:

  \040   is another way of writing a space
  \40    is the same, provided there are fewer than 40
            previous capturing subpatterns
  \7     is always a back reference
  \11    might be a back reference, or another way of
            writing a tab
  \011   is always a tab
  \0113  is a tab followed by the character "3"
  \113   might be a back reference, otherwise the
            character with octal code 113
  \377   might be a back reference, otherwise
            the byte consisting entirely of 1 bits
  \81    is either a back reference, or a binary zero
            followed by the two characters "8" and "1"

Note that octal values of 100 or greater must not be introduced by a leading zero, because no more than three octal digits are ever read.

All the sequences that define a single byte value or a single UTF-8 character (in UTF-8 mode) can be used both inside and outside character classes. In addition, inside a character class, the sequence \b is interpreted as the backspace character (hex 08). Outside a character class it has a different meaning (see below).

The third use of backslash is for specifying generic character types:

  \d     any decimal digit
  \D     any character that is not a decimal digit
  \s     any whitespace character
  \S     any character that is not a whitespace character
  \w     any "word" character
  \W     any "non-word" character

Each pair of escape sequences partitions the complete set of characters into two disjoint sets. Any given character matches one, and only one, of each pair.

In UTF-8 mode, characters with values greater than 255 never match \d, \s, or \w, and always match \D, \S, and \W.

For compatibility with Perl, \s does not match the VT character (code 11). This makes it different from the the POSIX "space" class. The \s characters are HT (9), LF (10), FF (12), CR (13), and space (32).

A "word" character is any letter or digit or the underscore character, that is, any character which can be part of a Perl "word". The definition of letters and digits is controlled by PCRE's character tables, and may vary if locale- specific matching is taking place (see "Locale support" in the pcreapi page). For example, in the "fr" (French) locale, some character codes greater than 128 are used for accented letters, and these are matched by \w.

These character type sequences can appear both inside and outside character classes. They each match one character of the appropriate type. If the current matching point is at the end of the subject string, all of them fail, since there is no character to match.

The fourth use of backslash is for certain simple assertions. An assertion specifies a condition that has to be met at a particular point in a match, without consuming any characters from the subject string. The use of subpatterns for more complicated assertions is described below. The backslashed assertions are

  \b     matches at a word boundary
  \B     matches when not at a word boundary
  \A     matches at start of subject
  \Z     matches at end of subject or before newline at end
  \z     matches at end of subject
  \G     matches at first matching position in subject

These assertions may not appear in character classes (but note that \b has a different meaning, namely the backspace character, inside a character class).

A word boundary is a position in the subject string where the current character and the previous character do not both match \w or \W (i.e. one matches \w and the other matches \W), or the start or end of the string if the first or last character matches \w, respectively.

The \A, \Z, and \z assertions differ from the traditional circumflex and dollar (described below) in that they only ever match at the very start and end of the subject string, whatever options are set. Thus, they are independent of multiline mode.

They are not affected by the PCRE_NOTBOL or PCRE_NOTEOL options. If the startoffset argument of pcre_exec() is non-zero, indicating that matching is to start at a point other than the beginning of the subject, \A can never match. The difference between \Z and \z is that \Z matches before a newline that is the last character of the string as well as at the end of the string, whereas \z matches only at the end.

The \G assertion is true only when the current matching position is at the start point of the match, as specified by the startoffset argument of pcre_exec(). It differs from \A when the value of startoffset is non-zero. By calling pcre_exec() multiple times with appropriate arguments, you can mimic Perl's /g option, and it is in this kind of implementation where \G can be useful.

Note, however, that PCRE's interpretation of \G, as the start of the current match, is subtly different from Perl's, which defines it as the end of the previous match. In Perl, these can be different when the previously matched string was empty. Because PCRE does just one match at a time, it cannot reproduce this behaviour.

If all the alternatives of a pattern begin with \G, the expression is anchored to the starting match position, and the "anchored" flag is set in the compiled regular expression.

CIRCUMFLEX AND DOLLAR

Outside a character class, in the default matching mode, the circumflex character is an assertion which is true only if the current matching point is at the start of the subject string. If the startoffset argument of pcre_exec() is non-zero, circumflex can never match if the PCRE_MULTILINE option is unset. Inside a character class, circumflex has an entirely different meaning (see below).

Circumflex need not be the first character of the pattern if a number of alternatives are involved, but it should be the first thing in each alternative in which it appears if the pattern is ever to match that branch. If all possible alternatives start with a circumflex, that is, if the pattern is constrained to match only at the start of the subject, it is said to be an "anchored" pattern. (There are also other constructs that can cause a pattern to be anchored.)

A dollar character is an assertion which is true only if the current matching point is at the end of the subject string, or immediately before a newline character that is the last character in the string (by default). Dollar need not be the last character of the pattern if a number of alternatives are involved, but it should be the last item in any branch in which it appears. Dollar has no special meaning in a character class.

The meaning of dollar can be changed so that it matches only at the very end of the string, by setting the PCRE_DOLLAR_ENDONLY option at compile time. This does not affect the \Z assertion.

The meanings of the circumflex and dollar characters are changed if the PCRE_MULTILINE option is set. When this is the case, they match immediately after and immediately before an internal newline character, respectively, in addition to matching at the start and end of the subject string. For example, the pattern /^abc$/ matches the subject string "def\nabc" in multiline mode, but not otherwise. Consequently, patterns that are anchored in single line mode because all branches start with ^ are not anchored in multiline mode, and a match for circumflex is possible when the startoffset argument of pcre_exec() is non-zero. The PCRE_DOLLAR_ENDONLY option is ignored if PCRE_MULTILINE is set.

Note that the sequences \A, \Z, and \z can be used to match the start and end of the subject in both modes, and if all branches of a pattern start with \A it is always anchored, whether PCRE_MULTILINE is set or not.

FULL STOP (PERIOD, DOT)

Outside a character class, a dot in the pattern matches any one character in the subject, including a non-printing character, but not (by default) newline. In UTF-8 mode, a dot matches any UTF-8 character, which might be more than one byte long, except (by default) for newline. If the PCRE_DOTALL option is set, dots match newlines as well. The handling of dot is entirely independent of the handling of circumflex and dollar, the only relationship being that they both involve newline characters. Dot has no special meaning in a character class.

MATCHING A SINGLE BYTE

Outside a character class, the escape sequence \C matches any one byte, both in and out of UTF-8 mode. Unlike a dot, it always matches a newline. The feature is provided in Perl in order to match individual bytes in UTF-8 mode. Because it breaks up UTF-8 characters into individual bytes, what remains in the string may be a malformed UTF-8 string. For this reason it is best avoided.

PCRE does not allow \C to appear in lookbehind assertions (see below), because in UTF-8 mode it makes it impossible to calculate the length of the lookbehind.

SQUARE BRACKETS

An opening square bracket introduces a character class, terminated by a closing square bracket. A closing square bracket on its own is not special. If a closing square bracket is required as a member of the class, it should be the first data character in the class (after an initial circumflex, if present) or escaped with a backslash.

A character class matches a single character in the subject. In UTF-8 mode, the character may occupy more than one byte. A matched character must be in the set of characters defined by the class, unless the first character in the class definition is a circumflex, in which case the subject character must not be in the set defined by the class. If a circumflex is actually required as a member of the class, ensure it is not the first character, or escape it with a backslash.

For example, the character class [aeiou] matches any lower case vowel, while [^aeiou] matches any character that is not a lower case vowel. Note that a circumflex is just a convenient notation for specifying the characters which are in the class by enumerating those that are not. It is not an assertion: it still consumes a character from the subject string, and fails if the current pointer is at the end of the string.

In UTF-8 mode, characters with values greater than 255 can be included in a class as a literal string of bytes, or by using the \x{ escaping mechanism.

When caseless matching is set, any letters in a class represent both their upper case and lower case versions, so for example, a caseless [aeiou] matches "A" as well as "a", and a caseless [^aeiou] does not match "A", whereas a caseful version would. PCRE does not support the concept of case for characters with values greater than 255.

The newline character is never treated in any special way in character classes, whatever the setting of the PCRE_DOTALL or PCRE_MULTILINE options is. A class such as [^a] will always match a newline.

The minus (hyphen) character can be used to specify a range of characters in a character class. For example, [d-m] matches any letter between d and m, inclusive. If a minus character is required in a class, it must be escaped with a backslash or appear in a position where it cannot be interpreted as indicating a range, typically as the first or last character in the class.

It is not possible to have the literal character "]" as the end character of a range. A pattern such as [W-]46] is interpreted as a class of two characters ("W" and "-") followed by a literal string "46]", so it would match "W46]" or "-46]". However, if the "]" is escaped with a backslash it is interpreted as the end of range, so [W-\]46] is interpreted as a single class containing a range followed by two separate characters. The octal or hexadecimal representation of "]" can also be used to end a range.

Ranges operate in the collating sequence of character values. They can also be used for characters specified numerically, for example [\000-\037]. In UTF-8 mode, ranges can include characters whose values are greater than 255, for example [\x{100}-\x{2ff}].

If a range that includes letters is used when caseless matching is set, it matches the letters in either case. For example, [W-c] is equivalent to [][\^_`wxyzabc], matched caselessly, and if character tables for the "fr" locale are in use, [\xc8-\xcb] matches accented E characters in both cases.

The character types \d, \D, \s, \S, \w, and \W may also appear in a character class, and add the characters that they match to the class. For example, [\dABCDEF] matches any hexadecimal digit. A circumflex can conveniently be used with the upper case character types to specify a more restricted set of characters than the matching lower case type. For example, the class [^\W_] matches any letter or digit, but not underscore.

All non-alphameric characters other than \, -, ^ (at the start) and the terminating ] are non-special in character classes, but it does no harm if they are escaped.

POSIX CHARACTER CLASSES

Perl supports the POSIX notation for character classes, which uses names enclosed by [: and :] within the enclosing square brackets. PCRE also supports this notation. For example,

  [01[:alpha:]%]

matches "0", "1", any alphabetic character, or "%". The supported class names are

  alnum    letters and digits
  alpha    letters
  ascii    character codes 0 - 127
  blank    space or tab only
  cntrl    control characters
  digit    decimal digits (same as \d)
  graph    printing characters, excluding space
  lower    lower case letters
  print    printing characters, including space
  punct    printing characters, excluding letters and digits
  space    white space (not quite the same as \s)
  upper    upper case letters
  word     "word" characters (same as \w)
  xdigit   hexadecimal digits

The "space" characters are HT (9), LF (10), VT (11), FF (12), CR (13), and space (32). Notice that this list includes the VT character (code 11). This makes "space" different to \s, which does not include VT (for Perl compatibility).

The name "word" is a Perl extension, and "blank" is a GNU extension from Perl 5.8. Another Perl extension is negation, which is indicated by a ^ character after the colon. For example,

  [12[:^digit:]]

matches "1", "2", or any non-digit. PCRE (and Perl) also recognize the POSIX syntax [.ch.] and [=ch=] where "ch" is a "collating element", but these are not supported, and an error is given if they are encountered.

In UTF-8 mode, characters with values greater than 255 do not match any of the POSIX character classes.

VERTICAL BAR

Vertical bar characters are used to separate alternative patterns. For example, the pattern

  gilbert|sullivan

matches either "gilbert" or "sullivan". Any number of alternatives may appear, and an empty alternative is permitted (matching the empty string). The matching process tries each alternative in turn, from left to right, and the first one that succeeds is used. If the alternatives are within a subpattern (defined below), "succeeds" means matching the rest of the main pattern as well as the alternative in the subpattern.

INTERNAL OPTION SETTING

The settings of the PCRE_CASELESS, PCRE_MULTILINE, PCRE_DOTALL, and PCRE_EXTENDED options can be changed from within the pattern by a sequence of Perl option letters enclosed between "(?" and ")". The option letters are

  i  for PCRE_CASELESS
  m  for PCRE_MULTILINE
  s  for PCRE_DOTALL
  x  for PCRE_EXTENDED

For example, (?im) sets caseless, multiline matching. It is also possible to unset these options by preceding the letter with a hyphen, and a combined setting and unsetting such as (?im-sx), which sets PCRE_CASELESS and PCRE_MULTILINE while unsetting PCRE_DOTALL and PCRE_EXTENDED, is also permitted. If a letter appears both before and after the hyphen, the option is unset.

When an option change occurs at top level (that is, not inside subpattern parentheses), the change applies to the remainder of the pattern that follows. If the change is placed right at the start of a pattern, PCRE extracts it into the global options (and it will therefore show up in data extracted by the pcre_fullinfo() function).

An option change within a subpattern affects only that part of the current pattern that follows it, so

  (a(?i)b)c

matches abc and aBc and no other strings (assuming PCRE_CASELESS is not used). By this means, options can be made to have different settings in different parts of the pattern. Any changes made in one alternative do carry on into subsequent branches within the same subpattern. For example,

  (a(?i)b|c)

matches "ab", "aB", "c", and "C", even though when matching "C" the first branch is abandoned before the option setting. This is because the effects of option settings happen at compile time. There would be some very weird behaviour otherwise.

The PCRE-specific options PCRE_UNGREEDY and PCRE_EXTRA can be changed in the same way as the Perl-compatible options by using the characters U and X respectively. The (?X) flag setting is special in that it must always occur earlier in the pattern than any of the additional features it turns on, even when it is at top level. It is best put at the start.

SUBPATTERNS

Subpatterns are delimited by parentheses (round brackets), which can be nested. Marking part of a pattern as a subpattern does two things:

1. It localizes a set of alternatives. For example, the pattern

  cat(aract|erpillar|)

matches one of the words "cat", "cataract", or "caterpillar". Without the parentheses, it would match "cataract", "erpillar" or the empty string.

2. It sets up the subpattern as a capturing subpattern (as defined above). When the whole pattern matches, that portion of the subject string that matched the subpattern is passed back to the caller via the ovector argument of pcre_exec(). Opening parentheses are counted from left to right (starting from 1) to obtain the numbers of the capturing subpatterns.

For example, if the string "the red king" is matched against the pattern

  the ((red|white) (king|queen))

the captured substrings are "red king", "red", and "king", and are numbered 1, 2, and 3, respectively.

The fact that plain parentheses fulfil two functions is not always helpful. There are often times when a grouping subpattern is required without a capturing requirement. If an opening parenthesis is followed by a question mark and a colon, the subpattern does not do any capturing, and is not counted when computing the number of any subsequent capturing subpatterns. For example, if the string "the white queen" is matched against the pattern

  the ((?:red|white) (king|queen))

the captured substrings are "white queen" and "queen", and are numbered 1 and 2. The maximum number of capturing subpatterns is 65535, and the maximum depth of nesting of all subpatterns, both capturing and non-capturing, is 200.

As a convenient shorthand, if any option settings are required at the start of a non-capturing subpattern, the option letters may appear between the "?" and the ":". Thus the two patterns

  (?i:saturday|sunday)
  (?:(?i)saturday|sunday)

match exactly the same set of strings. Because alternative branches are tried from left to right, and options are not reset until the end of the subpattern is reached, an option setting in one branch does affect subsequent branches, so the above patterns match "SUNDAY" as well as "Saturday".

NAMED SUBPATTERNS

Identifying capturing parentheses by number is simple, but it can be very hard to keep track of the numbers in complicated regular expressions. Furthermore, if an expression is modified, the numbers may change. To help with the difficulty, PCRE supports the naming of subpatterns, something that Perl does not provide. The Python syntax (?P<name>...) is used. Names consist of alphanumeric characters and underscores, and must be unique within a pattern.

Named capturing parentheses are still allocated numbers as well as names. The PCRE API provides function calls for extracting the name-to-number translation table from a compiled pattern. For further details see the pcreapi documentation.

REPETITION

Repetition is specified by quantifiers, which can follow any of the following items:

  a literal data character
  the . metacharacter
  the \C escape sequence
  escapes such as \d that match single characters
  a character class
  a back reference (see next section)
  a parenthesized subpattern (unless it is an assertion)

The general repetition quantifier specifies a minimum and maximum number of permitted matches, by giving the two numbers in curly brackets (braces), separated by a comma. The numbers must be less than 65536, and the first must be less than or equal to the second. For example:

  z{2,4}

matches "zz", "zzz", or "zzzz". A closing brace on its own is not a special character. If the second number is omitted, but the comma is present, there is no upper limit; if the second number and the comma are both omitted, the quantifier specifies an exact number of required matches. Thus

  [aeiou]{3,}

matches at least 3 successive vowels, but may match many more, while

  \d{8}

matches exactly 8 digits. An opening curly bracket that appears in a position where a quantifier is not allowed, or one that does not match the syntax of a quantifier, is taken as a literal character. For example, {,6} is not a quantifier, but a literal string of four characters.

In UTF-8 mode, quantifiers apply to UTF-8 characters rather than to individual bytes. Thus, for example, \x{100}{2} matches two UTF-8 characters, each of which is represented by a two-byte sequence.

The quantifier {0} is permitted, causing the expression to behave as if the previous item and the quantifier were not present.

For convenience (and historical compatibility) the three most common quantifiers have single-character abbreviations:

  *    is equivalent to {0,}
  +    is equivalent to {1,}
  ?    is equivalent to {0,1}

It is possible to construct infinite loops by following a subpattern that can match no characters with a quantifier that has no upper limit, for example:

  (a?)*

Earlier versions of Perl and PCRE used to give an error at compile time for such patterns. However, because there are cases where this can be useful, such patterns are now accepted, but if any repetition of the subpattern does in fact match no characters, the loop is forcibly broken.

By default, the quantifiers are "greedy", that is, they match as much as possible (up to the maximum number of permitted times), without causing the rest of the pattern to fail. The classic example of where this gives problems is in trying to match comments in C programs. These appear between the sequences /* and */ and within the sequence, individual * and / characters may appear. An attempt to match C comments by applying the pattern

  /\*.*\*/

to the string

  /* first command */  not comment  /* second comment */

fails, because it matches the entire string owing to the greediness of the .* item.

However, if a quantifier is followed by a question mark, it ceases to be greedy, and instead matches the minimum number of times possible, so the pattern

  /\*.*?\*/

does the right thing with the C comments. The meaning of the various quantifiers is not otherwise changed, just the preferred number of matches. Do not confuse this use of question mark with its use as a quantifier in its own right. Because it has two uses, it can sometimes appear doubled, as in

  \d??\d

which matches one digit by preference, but can match two if that is the only way the rest of the pattern matches.

If the PCRE_UNGREEDY option is set (an option which is not available in Perl), the quantifiers are not greedy by default, but individual ones can be made greedy by following them with a question mark. In other words, it inverts the default behaviour.

When a parenthesized subpattern is quantified with a minimum repeat count that is greater than 1 or with a limited maximum, more store is required for the compiled pattern, in proportion to the size of the minimum or maximum.

If a pattern starts with .* or .{0,} and the PCRE_DOTALL option (equivalent to Perl's /s) is set, thus allowing the . to match newlines, the pattern is implicitly anchored, because whatever follows will be tried against every character position in the subject string, so there is no point in retrying the overall match at any position after the first. PCRE normally treats such a pattern as though it were preceded by \A.

In cases where it is known that the subject string contains no newlines, it is worth setting PCRE_DOTALL in order to obtain this optimization, or alternatively using ^ to indicate anchoring explicitly.

However, there is one situation where the optimization cannot be used. When .* is inside capturing parentheses that are the subject of a backreference elsewhere in the pattern, a match at the start may fail, and a later one succeed. Consider, for example:

  (.*)abc\1

If the subject is "xyz123abc123" the match point is the fourth character. For this reason, such a pattern is not implicitly anchored.

When a capturing subpattern is repeated, the value captured is the substring that matched the final iteration. For example, after

  (tweedle[dume]{3}\s*)+

has matched "tweedledum tweedledee" the value of the captured substring is "tweedledee". However, if there are nested capturing subpatterns, the corresponding captured values may have been set in previous iterations. For example, after

  /(a|(b))+/

matches "aba" the value of the second captured substring is "b".

ATOMIC GROUPING AND POSSESSIVE QUANTIFIERS

With both maximizing and minimizing repetition, failure of what follows normally causes the repeated item to be re-evaluated to see if a different number of repeats allows the rest of the pattern to match. Sometimes it is useful to prevent this, either to change the nature of the match, or to cause it fail earlier than it otherwise might, when the author of the pattern knows there is no point in carrying on.

Consider, for example, the pattern \d+foo when applied to the subject line

  123456bar

After matching all 6 digits and then failing to match "foo", the normal action of the matcher is to try again with only 5 digits matching the \d+ item, and then with 4, and so on, before ultimately failing. "Atomic grouping" (a term taken from Jeffrey Friedl's book) provides the means for specifying that once a subpattern has matched, it is not to be re-evaluated in this way.

If we use atomic grouping for the previous example, the matcher would give up immediately on failing to match "foo" the first time. The notation is a kind of special parenthesis, starting with (?> as in this example:

  (?>\d+)bar

This kind of parenthesis "locks up" the part of the pattern it contains once it has matched, and a failure further into the pattern is prevented from backtracking into it. Backtracking past it to previous items, however, works as normal.

An alternative description is that a subpattern of this type matches the string of characters that an identical standalone pattern would match, if anchored at the current point in the subject string.

Atomic grouping subpatterns are not capturing subpatterns. Simple cases such as the above example can be thought of as a maximizing repeat that must swallow everything it can. So, while both \d+ and \d+? are prepared to adjust the number of digits they match in order to make the rest of the pattern match, (?>\d+) can only match an entire sequence of digits.

Atomic groups in general can of course contain arbitrarily complicated subpatterns, and can be nested. However, when the subpattern for an atomic group is just a single repeated item, as in the example above, a simpler notation, called a "possessive quantifier" can be used. This consists of an additional + character following a quantifier. Using this notation, the previous example can be rewritten as

  \d++bar

Possessive quantifiers are always greedy; the setting of the PCRE_UNGREEDY option is ignored. They are a convenient notation for the simpler forms of atomic group. However, there is no difference in the meaning or processing of a possessive quantifier and the equivalent atomic group.

The possessive quantifier syntax is an extension to the Perl syntax. It originates in Sun's Java package.

When a pattern contains an unlimited repeat inside a subpattern that can itself be repeated an unlimited number of times, the use of an atomic group is the only way to avoid some failing matches taking a very long time indeed. The pattern

  (\D+|<\d+>)*[!?]

matches an unlimited number of substrings that either consist of non-digits, or digits enclosed in <>, followed by either ! or ?. When it matches, it runs quickly. However, if it is applied to

  aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

it takes a long time before reporting failure. This is because the string can be divided between the two repeats in a large number of ways, and all have to be tried. (The example used [!?] rather than a single character at the end, because both PCRE and Perl have an optimization that allows for fast failure when a single character is used. They remember the last single character that is required for a match, and fail early if it is not present in the string.) If the pattern is changed to

  ((?>\D+)|<\d+>)*[!?]

sequences of non-digits cannot be broken, and failure happens quickly.

BACK REFERENCES

Outside a character class, a backslash followed by a digit greater than 0 (and possibly further digits) is a back reference to a capturing subpattern earlier (that is, to its left) in the pattern, provided there have been that many previous capturing left parentheses.

However, if the decimal number following the backslash is less than 10, it is always taken as a back reference, and causes an error only if there are not that many capturing left parentheses in the entire pattern. In other words, the parentheses that are referenced need not be to the left of the reference for numbers less than 10. See the section entitled "Backslash" above for further details of the handling of digits following a backslash.

A back reference matches whatever actually matched the capturing subpattern in the current subject string, rather than anything matching the subpattern itself (see "Subpatterns as subroutines" below for a way of doing that). So the pattern

  (sens|respons)e and \1ibility

matches "sense and sensibility" and "response and responsibility", but not "sense and responsibility". If caseful matching is in force at the time of the back reference, the case of letters is relevant. For example,

  ((?i)rah)\s+\1

matches "rah rah" and "RAH RAH", but not "RAH rah", even though the original capturing subpattern is matched caselessly.

Back references to named subpatterns use the Python syntax (?P=name). We could rewrite the above example as follows:

  (?<p1>(?i)rah)\s+(?P=p1)

There may be more than one back reference to the same subpattern. If a subpattern has not actually been used in a particular match, any back references to it always fail. For example, the pattern

  (a|(bc))\2

always fails if it starts to match "a" rather than "bc". Because there may be many capturing parentheses in a pattern, all digits following the backslash are taken as part of a potential back reference number. If the pattern continues with a digit character, some delimiter must be used to terminate the back reference. If the PCRE_EXTENDED option is set, this can be whitespace. Otherwise an empty comment can be used.

A back reference that occurs inside the parentheses to which it refers fails when the subpattern is first used, so, for example, (a\1) never matches. However, such references can be useful inside repeated subpatterns. For example, the pattern

  (a|b\1)+

matches any number of "a"s and also "aba", "ababbaa" etc. At each iteration of the subpattern, the back reference matches the character string corresponding to the previous iteration. In order for this to work, the pattern must be such that the first iteration does not need to match the back reference. This can be done using alternation, as in the example above, or by a quantifier with a minimum of zero.

ASSERTIONS

An assertion is a test on the characters following or preceding the current matching point that does not actually consume any characters. The simple assertions coded as \b, \B, \A, \G, \Z, \z, ^ and $ are described above. More complicated assertions are coded as subpatterns. There are two kinds: those that look ahead of the current position in the subject string, and those that look behind it.

An assertion subpattern is matched in the normal way, except that it does not cause the current matching position to be changed. Lookahead assertions start with (?= for positive assertions and (?! for negative assertions. For example,

  \w+(?=;)

matches a word followed by a semicolon, but does not include the semicolon in the match, and

  foo(?!bar)

matches any occurrence of "foo" that is not followed by "bar". Note that the apparently similar pattern

  (?!foo)bar

does not find an occurrence of "bar" that is preceded by something other than "foo"; it finds any occurrence of "bar" whatsoever, because the assertion (?!foo) is always true when the next three characters are "bar". A lookbehind assertion is needed to achieve this effect.

If you want to force a matching failure at some point in a pattern, the most convenient way to do it is with (?!) because an empty string always matches, so an assertion that requires there not to be an empty string must always fail.

Lookbehind assertions start with (?<= for positive assertions and (?<! for negative assertions. For example,

  (?<!foo)bar

does find an occurrence of "bar" that is not preceded by "foo". The contents of a lookbehind assertion are restricted such that all the strings it matches must have a fixed length. However, if there are several alternatives, they do not all have to have the same fixed length. Thus

  (?<=bullock|donkey)

is permitted, but

  (?<!dogs?|cats?)

causes an error at compile time. Branches that match different length strings are permitted only at the top level of a lookbehind assertion. This is an extension compared with Perl (at least for 5.8), which requires all branches to match the same length of string. An assertion such as

  (?<=ab(c|de))

is not permitted, because its single top-level branch can match two different lengths, but it is acceptable if rewritten to use two top-level branches:

  (?<=abc|abde)

The implementation of lookbehind assertions is, for each alternative, to temporarily move the current position back by the fixed width and then try to match. If there are insufficient characters before the current position, the match is deemed to fail.

PCRE does not allow the \C escape (which matches a single byte in UTF-8 mode) to appear in lookbehind assertions, because it makes it impossible to calculate the length of the lookbehind.

Atomic groups can be used in conjunction with lookbehind assertions to specify efficient matching at the end of the subject string. Consider a simple pattern such as

  abcd$

when applied to a long string that does not match. Because matching proceeds from left to right, PCRE will look for each "a" in the subject and then see if what follows matches the rest of the pattern. If the pattern is specified as

  ^.*abcd$

the initial .* matches the entire string at first, but when this fails (because there is no following "a"), it backtracks to match all but the last character, then all but the last two characters, and so on. Once again the search for "a" covers the entire string, from right to left, so we are no better off. However, if the pattern is written as

  ^(?>.*)(?<=abcd)

or, equivalently,

  ^.*+(?<=abcd)

there can be no backtracking for the .* item; it can match only the entire string. The subsequent lookbehind assertion does a single test on the last four characters. If it fails, the match fails immediately. For long strings, this approach makes a significant difference to the processing time.

Several assertions (of any sort) may occur in succession. For example,

  (?<=\d{3})(?<!999)foo

matches "foo" preceded by three digits that are not "999". Notice that each of the assertions is applied independently at the same point in the subject string. First there is a check that the previous three characters are all digits, and then there is a check that the same three characters are not "999". This pattern does not match "foo" preceded by six characters, the first of which are digits and the last three of which are not "999". For example, it doesn't match "123abcfoo". A pattern to do that is

  (?<=\d{3}...)(?<!999)foo

This time the first assertion looks at the preceding six characters, checking that the first three are digits, and then the second assertion checks that the preceding three characters are not "999".

Assertions can be nested in any combination. For example,

  (?<=(?<!foo)bar)baz

matches an occurrence of "baz" that is preceded by "bar" which in turn is not preceded by "foo", while

  (?<=\d{3}(?!999)...)foo

is another pattern which matches "foo" preceded by three digits and any three characters that are not "999".

Assertion subpatterns are not capturing subpatterns, and may not be repeated, because it makes no sense to assert the same thing several times. If any kind of assertion contains capturing subpatterns within it, these are counted for the purposes of numbering the capturing subpatterns in the whole pattern. However, substring capturing is carried out only for positive assertions, because it does not make sense for negative assertions.

CONDITIONAL SUBPATTERNS

It is possible to cause the matching process to obey a subpattern conditionally or to choose between two alternative subpatterns, depending on the result of an assertion, or whether a previous capturing subpattern matched or not. The two possible forms of conditional subpattern are

  (?(condition)yes-pattern)
  (?(condition)yes-pattern|no-pattern)

If the condition is satisfied, the yes-pattern is used; otherwise the no-pattern (if present) is used. If there are more than two alternatives in the subpattern, a compile-time error occurs.

There are three kinds of condition. If the text between the parentheses consists of a sequence of digits, the condition is satisfied if the capturing subpattern of that number has previously matched. The number must be greater than zero. Consider the following pattern, which contains non-significant white space to make it more readable (assume the PCRE_EXTENDED option) and to divide it into three parts for ease of discussion:

  ( \( )?    [^()]+    (?(1) \) )

The first part matches an optional opening parenthesis, and if that character is present, sets it as the first captured substring. The second part matches one or more characters that are not parentheses. The third part is a conditional subpattern that tests whether the first set of parentheses matched or not. If they did, that is, if subject started with an opening parenthesis, the condition is true, and so the yes-pattern is executed and a closing parenthesis is required. Otherwise, since no-pattern is not present, the subpattern matches nothing. In other words, this pattern matches a sequence of non-parentheses, optionally enclosed in parentheses.

If the condition is the string (R), it is satisfied if a recursive call to the pattern or subpattern has been made. At "top level", the condition is false. This is a PCRE extension. Recursive patterns are described in the next section.

If the condition is not a sequence of digits or (R), it must be an assertion. This may be a positive or negative lookahead or lookbehind assertion. Consider this pattern, again containing non-significant white space, and with the two alternatives on the second line:

  (?(?=[^a-z]*[a-z])
  \d{2}-[a-z]{3}-\d{2}  |  \d{2}-\d{2}-\d{2} )

The condition is a positive lookahead assertion that matches an optional sequence of non-letters followed by a letter. In other words, it tests for the presence of at least one letter in the subject. If a letter is found, the subject is matched against the first alternative; otherwise it is matched against the second. This pattern matches strings in one of the two forms dd-aaa-dd or dd-dd-dd, where aaa are letters and dd are digits.

COMMENTS

The sequence (?# marks the start of a comment which continues up to the next closing parenthesis. Nested parentheses are not permitted. The characters that make up a comment play no part in the pattern matching at all.

If the PCRE_EXTENDED option is set, an unescaped # character outside a character class introduces a comment that continues up to the next newline character in the pattern.

RECURSIVE PATTERNS

Consider the problem of matching a string in parentheses, allowing for unlimited nested parentheses. Without the use of recursion, the best that can be done is to use a pattern that matches up to some fixed depth of nesting. It is not possible to handle an arbitrary nesting depth. Perl has provided an experimental facility that allows regular expressions to recurse (amongst other things). It does this by interpolating Perl code in the expression at run time, and the code can refer to the expression itself. A Perl pattern to solve the parentheses problem can be created like this:

  $re = qr{\( (?: (?>[^()]+) | (?p{$re}) )* \)}x;

The (?p{...}) item interpolates Perl code at run time, and in this case refers recursively to the pattern in which it appears. Obviously, PCRE cannot support the interpolation of Perl code. Instead, it supports some special syntax for recursion of the entire pattern, and also for individual subpattern recursion.

The special item that consists of (? followed by a number greater than zero and a closing parenthesis is a recursive call of the subpattern of the given number, provided that it occurs inside that subpattern. (If not, it is a "subroutine" call, which is described in the next section.) The special item (?R) is a recursive call of the entire regular expression.

For example, this PCRE pattern solves the nested parentheses problem (assume the PCRE_EXTENDED option is set so that white space is ignored):

  \( ( (?>[^()]+) | (?R) )* \)

First it matches an opening parenthesis. Then it matches any number of substrings which can either be a sequence of non-parentheses, or a recursive match of the pattern itself (that is a correctly parenthesized substring). Finally there is a closing parenthesis.

If this were part of a larger pattern, you would not want to recurse the entire pattern, so instead you could use this:

  ( \( ( (?>[^()]+) | (?1) )* \) )

We have put the pattern into parentheses, and caused the recursion to refer to them instead of the whole pattern. In a larger pattern, keeping track of parenthesis numbers can be tricky. It may be more convenient to use named parentheses instead. For this, PCRE uses (?P>name), which is an extension to the Python syntax that PCRE uses for named parentheses (Perl does not provide named parentheses). We could rewrite the above example as follows:

  (?<pn> \( ( (?>[^()]+) | (?P>pn) )* \) )

This particular example pattern contains nested unlimited repeats, and so the use of atomic grouping for matching strings of non-parentheses is important when applying the pattern to strings that do not match. For example, when this pattern is applied to

  (aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa()

it yields "no match" quickly. However, if atomic grouping is not used, the match runs for a very long time indeed because there are so many different ways the + and * repeats can carve up the subject, and all have to be tested before failure can be reported.

At the end of a match, the values set for any capturing subpatterns are those from the outermost level of the recursion at which the subpattern value is set. If you want to obtain intermediate values, a callout function can be used (see below and the pcrecallout documentation). If the pattern above is matched against

  (ab(cd)ef)

the value for the capturing parentheses is "ef", which is the last value taken on at the top level. If additional parentheses are added, giving

  \( ( ( (?>[^()]+) | (?R) )* ) \)
     ^                        ^
     ^                        ^

the string they capture is "ab(cd)ef", the contents of the top level parentheses. If there are more than 15 capturing parentheses in a pattern, PCRE has to obtain extra memory to store data during a recursion, which it does by using pcre_malloc, freeing it via pcre_free afterwards. If no memory can be obtained, the match fails with the PCRE_ERROR_NOMEMORY error.

Do not confuse the (?R) item with the condition (R), which tests for recursion. Consider this pattern, which matches text in angle brackets, allowing for arbitrary nesting. Only digits are allowed in nested brackets (that is, when recursing), whereas any characters are permitted at the outer level.

  < (?: (?(R) \d++  | [^<>]*+) | (?R)) * >

In this pattern, (?(R) is the start of a conditional subpattern, with two different alternatives for the recursive and non-recursive cases. The (?R) item is the actual recursive call.

SUBPATTERNS AS SUBROUTINES

If the syntax for a recursive subpattern reference (either by number or by name) is used outside the parentheses to which it refers, it operates like a subroutine in a programming language. An earlier example pointed out that the pattern

  (sens|respons)e and \1ibility

matches "sense and sensibility" and "response and responsibility", but not "sense and responsibility". If instead the pattern

  (sens|respons)e and (?1)ibility

is used, it does match "sense and responsibility" as well as the other two strings. Such references must, however, follow the subpattern to which they refer.

CALLOUTS

Perl has a feature whereby using the sequence (?{...}) causes arbitrary Perl code to be obeyed in the middle of matching a regular expression. This makes it possible, amongst other things, to extract different substrings that match the same pair of parentheses when there is a repetition.

PCRE provides a similar feature, but of course it cannot obey arbitrary Perl code. The feature is called "callout". The caller of PCRE provides an external function by putting its entry point in the global variable pcre_callout. By default, this variable contains NULL, which disables all calling out.

Within a regular expression, (?C) indicates the points at which the external function is to be called. If you want to identify different callout points, you can put a number less than 256 after the letter C. The default value is zero. For example, this pattern has two callout points:

  (?C1)\dabc(?C2)def

During matching, when PCRE reaches a callout point (and pcre_callout is set), the external function is called. It is provided with the number of the callout, and, optionally, one item of data originally supplied by the caller of pcre_exec(). The callout function may cause matching to backtrack, or to fail altogether. A complete description of the interface to the callout function is given in the pcrecallout documentation.

Last updated: 03 February 2004
Copyright © 1997-2004 University of Cambridge.

UTF-8 SUPPORT

Starting at release 3.3, PCRE has had some support for character strings encoded in the UTF-8 format. For release 4.0 this has been greatly extended to cover most common requirements.

In order process UTF-8 strings, you must build PCRE to include UTF-8 support in the code, and, in addition, you must call pcre_compile() with the PCRE_UTF8 option flag. When you do this, both the pattern and any subject strings that are matched against it are treated as UTF-8 strings instead of just strings of bytes.

If you compile PCRE with UTF-8 support, but do not use it at run time, the library will be a bit bigger, but the additional run time overhead is limited to testing the PCRE_UTF8 flag in several places, so should not be very large.

The following comments apply when PCRE is running in UTF-8 mode:

1. PCRE assumes that the strings it is given contain valid UTF-8 codes. It does not diagnose invalid UTF-8 strings. If you pass invalid UTF-8 strings to PCRE, the results are undefined.

2. In a pattern, the escape sequence \x{...}, where the contents of the braces is a string of hexadecimal digits, is interpreted as a UTF-8 character whose code number is the given hexadecimal number, for example: \x{1234}. If a non-hexadecimal digit appears between the braces, the item is not recognized. This escape sequence can be used either as a literal, or within a character class.

3. The original hexadecimal escape sequence, \xhh, matches a two-byte UTF-8 character if the value is greater than 127.

4. Repeat quantifiers apply to complete UTF-8 characters, not to individual bytes, for example: \x{100}{3}.

5. The dot metacharacter matches one UTF-8 character instead of a single byte.

6. The escape sequence \C can be used to match a single byte in UTF-8 mode, but its use can lead to some strange effects.

7. The character escapes \b, \B, \d, \D, \s, \S, \w, and \W correctly test characters of any code value, but the characters that PCRE recognizes as digits, spaces, or word characters remain the same set as before, all with values less than 256.

8. Case-insensitive matching applies only to characters whose values are less than 256. PCRE does not support the notion of "case" for higher-valued characters.

9. PCRE does not support the use of Unicode tables and properties or the Perl escapes \p, \P, and \X.

DIFFERENCES FROM PERL

This document describes the differences in the ways that PCRE and Perl handle regular expressions. The differences described here are with respect to Perl 5.8.

1. PCRE does not allow repeat quantifiers on lookahead assertions. Perl permits them, but they do not mean what you might think. For example, (?!a){3} does not assert that the next three characters are not "a". It just asserts that the next character is not "a" three times.

2. Capturing subpatterns that occur inside negative lookahead assertions are counted, but their entries in the offsets vector are never set. Perl sets its numerical variables from any such patterns that are matched before the assertion fails to match something (thereby succeeding), but only if the negative lookahead assertion contains just one branch.

3. Though binary zero characters are supported in the subject string, they are not allowed in a pattern string because it is passed as a normal C string, terminated by zero. The escape sequence "\0" can be used in the pattern to represent a binary zero.

4. The following Perl escape sequences are not supported: \l, \u, \L, \U, \P, \p, \N, and \X. In fact these are implemented by Perl's general string-handling and are not part of its pattern matching engine. If any of these are encountered by PCRE, an error is generated.

5. PCRE does support the \Q...\E escape for quoting substrings. Characters in between are treated as literals. This is slightly different from Perl in that $ and @ are also handled as literals inside the quotes. In Perl, they cause variable interpolation (but of course PCRE does not have variables). Note the following examples:

    Pattern            PCRE matches      Perl matches

    \Qabc$xyz\E        abc$xyz           abc followed by the
                                           contents of $xyz
    \Qabc\$xyz\E       abc\$xyz          abc\$xyz
    \Qabc\E\$\Qxyz\E   abc$xyz           abc$xyz

In PCRE, the \Q...\E mechanism is not recognized inside a character class.

8. Fairly obviously, PCRE does not support the (?{code}) and (?p{code}) constructions. However, there is some experimental support for recursive patterns using the non-Perl items (?R), (?number) and (?P>name). Also, the PCRE "callout" feature allows an external function to be called during pattern matching.

9. There are some differences that are concerned with the settings of captured strings when part of a pattern is repeated. For example, matching "aba" against the pattern /^(a(b)?)+$/ in Perl leaves $2 unset, but in PCRE it is set to "b".

10. PCRE provides some extensions to the Perl regular expression facilities:

(a) Although lookbehind assertions must match fixed length strings, each alternative branch of a lookbehind assertion can match a different length of string. Perl requires them all to have the same length.

(b) If PCRE_DOLLAR_ENDONLY is set and PCRE_MULTILINE is not set, the $ meta-character matches only at the very end of the string.

© If PCRE_EXTRA is set, a backslash followed by a letter with no special meaning is faulted.

(d) If PCRE_UNGREEDY is set, the greediness of the repetition quantifiers is inverted, that is, by default they are not greedy, but if followed by a question mark they are.

(e) PCRE_ANCHORED can be used to force a pattern to be tried only at the first matching position in the subject string.

(f) The PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, and PCRE_NO_AUTO_CAPTURE options for pcre_exec() have no Perl equivalents.

(g) The (?R), (?number), and (?P>name) constructs allows for recursive pattern matching (Perl can do this using the (?p{code}) construct, which PCRE cannot support.)

(h) PCRE supports named capturing substrings, using the Python syntax.

(i) PCRE supports the possessive quantifier "++" syntax, taken from Sun's Java package.

(j) The (R) condition, for testing recursion, is a PCRE extension.

(k) The callout facility is PCRE-specific.

Last updated: 03 February 2004
Copyright © 1997-2004 University of Cambridge.


EasyPatterns

 EasyPatterns

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up
EasyPatterns Reference

EasyPatterns™ are a revolutionary new way to describe text patterns that is easy to understand, easy to use, and still just as powerful as the old-hard to understand Perl-style or grep-style pattern matching languages.

What is a Pattern?

A pattern describes text in a general way. Searching based on literal text yields only one match, whereas searching with a pattern can match a whole class of text that shares the specified characteristics. Phone numbers (in North America) are a perfect example: three groups of digits separated by various punctuation characters. Here's a simple Text Machine pattern that will find most phone numbers:
[3 digits, punctuation, 3 digits, punctuation, 4 digits]

Let's suppose that you have a list of addresses phone numbers gathered from different sources. The phone numbers are likely to be in many formats: some with / and -, some with periods, etc. To convert them to a consistent format you need to keep the three groups of digits and insert new punctuation in between. With TextPipe's EasyPatterns, this is as easy as
Search for: "[punctuation (3 digits)1 punctuation (3 digits)2 punctuation (4 digits)3]"
Replace with: "$1-$2-$3"

This section introduces all of the essential elements of EasyPattern patterns. If you are new to pattern-matching, there's no reason you have to read the entire section at once. Feel free to stop after any section and continue with the next 2 sections or spend time reviewing example. Also, don't worry if you don't immediately understand every point. The only way to learn pattern matching is to create your own patterns, including the inevitable trial and error required to get just the right match. TextPipe's trial run area was designed with this in mind.

Throughout this text and the reference section, double quotes are used around both the pattern and the example text that it would match. The double quotes should *NOT* be entered in TextPipe.

Basics

Unlike other pattern matching languages, EasyPatterns use plain English keywords such as "letter" and "digit" to define text patterns. At the most basic level, each keyword appears in [square brackets], e.g.
"[letter][digit]" « matches "a0" ... "a9", "b0" ... "b9", etc.

Why use brackets? Patterns can also include literal text; brackets distinguish that from the pattern specification:
"abc[digit]" « matches "abc0", "abc1" ... "abc9"
"Z[letter][letter]" « matches "Zoo", "Zed", etc.
"whatever" « keywords are not required, EasyPattern can search just for literal text

In addition to simple keywords, the brackets can also contain expressions. The most common expressions indicate repetition, e.g.
"[optional digit]" « 0 or 1
"[oneOrMore digits]" « 1, 2 or many (no spaces are allowed in "oneOrMore")
"[1+ digits]" « same meaning, different notation
"[3+ digits]" « 3 or more
"[3 digits]" « exactly 3
"[3 to 5 digits]" « 3, 4 or 5

Simple keywords and expressions can be combined, e.g. here's a pattern that matches many North American telephone numbers:
"[3 digits][punctuation][3 digits][punctuation][4 digits]"

That's a lot of brackets! The above pattern works, but EasyPattern also allows multiple keywords and expressions to be combined with commas, e.g.
"[3 digits, punctuation, 3 digits, punctuation, 4 digits]"

In fact, even the commas are optional, though you may find they make the pattern easier to read.
"[3 digits punctuation 3 digits punctuation 4 digits]"

Many patterns require mixing keywords with literal text, e.g. to find telephone numbers in a certain local exchange:
"978[punctuation]692[punctuation, 4 digits]"

That pattern works fine; EasyPattern also lets you include the literal text inside the bracketed expression using single quotes:
"['978', punctuation, '692', punctuation, 4 digits]"

Character Sets

One of the fundamental ideas in pattern matching is that each character belongs to one or more character sets. For example, "a" is a member of the set of letters, represented by the keyword "[letter]". "2" is a member of the set "[digit]", etc. In EasyPattern, character sets can be combined with "or". For example, both "a" and "2" are members of the combined set "[letter or digit]". You can combine any number of sets with "or", e.g. "[letter or punctuation or symbol or whitespace]" is nearly everything except digits.

You can also use "or" to combine individual characters into a set, e.g. "[space or tab]". See the EasyPattern Reference for many additional whitespace options.

Tip: Parentheses are not required. For example, "[1+ space or tab]" and "[1+ (space or tab)]" have the same meaning.

The keyword "[char]" or "[character]" represents the set of all characters, including letters, digits, punctuation, symbols, space, tab, return, linefeed, formfeed and control characters. (NULL is not included; see EasyPattern Reference for details and a workaround.) Sometimes you want to match every character in a paragraph, i.e. every character except return (or formfeed). EasyPattern has a special keyword "[paragraphChar]" for this situation. For example, given the text "abc" then a return then "123":
"[1+ char]" « matches all 7 characters
"[1+ paragraphChar]" « matches the first 3 characters

Tip: Look up "[paragraphChar]" in the EasyPattern Reference; you'll find a whole family of keywords for words, columns, lines & paragraphs.

Sometimes it's easier to specify the set that you don't want rather than listing everything you want:
"[not letter]" « match anything except a letter "[not letter or digit]" « match anything except a letter or digit "[1+ not letter or digit]" « matches a sequence of characters Tip: Since "not" is processed after the character set use of "or", parentheses are still optional. For example, "[not space or tab]" and "[not (space or tab)]" have the same meaning.

Alternatives

One of the most powerful features of EasyPattern isn't related to patterns per se: the ability to find a match from among two or more choices. Some examples:
"this[or]that"
"['this' or 'that']"
"EasyPattern[or]Player"
"StarTrek[or]StarWars[or]OSA[or]scripting"

Tip: If you include an alternative in a larger pattern, you must include parentheses:
"[digit ('this' or 'that')]"

The keyword "or" has two uses: combining character sets and listing alternatives. The differences are sometimes subtle though will become more clear with experience. See the EasyPattern Reference for details.

Groups

EasyPattern lets you group any part of a pattern with parentheses. Groups can also be identified with a number to capture the text that the group matched for later use, either in a replacement string, or in a later part of the pattern, e.g. [( ... )1]. The number given is irrelevant – it simply serves to mark the parentheses as a group.

There are many reasons to group parts of a pattern:

  • Apply quantity or repetition to a group, e.g. [1+ (letter, digit)] will match "r2d2"
  • Specify one of many alternatives with "or"
  • Mark logical groups to make the pattern easier to read
  • Extract a certain portion of a pattern (where the full pattern is required to provide context)
  • Replace certain portions of a pattern, including changing the order

For example, you could change phone numbers from many different formats to your preferred format, searching for:
"[punctuation (3 digits)1 punctuation (3 digits)2 punctuation (4 digits)3]"

and replacing with:
"$1-$2-$3"

Tip: Parentheses groups can be nested to any level, e.g. [1+ ((letter, digit) or (digit, letter))] will match "r2d2", "r22d", "2rd2", "2r2d", etc.

Case in Keywords

EasyPattern keywords are not allowed to contain spaces, and are not case sensitive; e.g. [uppercaseLetter], [UppercaseLetter], [UPPERCASELETTER], [UppErcAsELEttEr] are equivalent.

Case in Literals

Although keywords are not case sensitive, the case sensitivity of literal text is controlled by TextPipe 'Match Case' option.

Patterns in Replace Strings

In the simplest replacement, you will find a pattern and replace it with literal text, e.g.
replace "*[or]_" with "-"

To add literal text before or after existing text, use $0 to represent the entire found text, e.g.
replace "-[space]" with "\t$0" -- insert a tab before list items

The phone number example shows a common use of [(...)#] and $# (where # is a number from 0-9); here's another example:
replace "[(column)1 tab (column)2]" with "$2\t$1" -- swap columns

What Next?

EasyPattern's core vocabulary was covered in this section. The best way to learn is by creating patterns for real-world tasks. When you have questions on the basics, refer back to this section. If you need a more detailed explanation of any topic or are looking for additional keywords to handle a new case, see the EasyPattern Reference.


EasyPatterns Reference

 EasyPatterns Reference

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Note: For the latest copy of this documentation, please see the complete EasyPatterns reference (online)

Literal Text, Keywords and Expressions

Notation: "..." (without the quotes) is any appropriate text, keyword or expression.

The entire pattern is a string, so it is enclosed in double-quotes. With a few exceptions, the quotation marks will not be shown in this section.

the simplest pattern is just literal text
... e.g. "abc"
EasyPattern keywords are enclosed in brackets
[...] e.g. "[letter]"
literal text & keywords can be combined
... [...] | e.g. "abc[digit]" matches "abc1", "abc2", etc.
[...] ... | e.g. "[digit]abc" matches "1abc", "2abc", etc.
... [...] ... | e.g. "[digit]abc[digit]" matches "1abc1", "1abc2", "2abc1", etc.
multiple keywords can appear together
[...][...] e.g. [letter][digit] matches "a1", "b1", etc.
[..., ...] | e.g. [letter, digit] instead of [letter][digit]
[... ...] | e.g. [letter digit] instead of [letter][digit]
  • Any "][" pair in a pattern (other than within a quoted string) can be replaced with a comma (followed by an optional space). Any comma (and optional space) in a pattern can be replaced by "][". The meaning is the same.
  • Any "][" pair or comma in a pattern (other than within a quoted string) can be replaced with a space (actually, one or more spaces), any space in a pattern can be replaced by a comma or "][". The meaning is the same.
literal text can be included inside a bracketed expression using single quotes
['literal'] | e.g. "['abc']" instead of "abc"
[... 'literal'] | e.g. "[digit, 'abc']" instead of "[digit]abc"
['literal' ...] | e.g. "['abc', digit]" instead of "abc[digit]"
[... 'literal' ...] | e.g. "[digit, 'abc', digit]" instead of "[digit]abc[digit]"
  • There is usually no difference in meaning between including literal text within a bracketed expression (in single quotes) and leaving literal text outside the brackets. The choice is a matter of individual preference. One exception: when using [not], only the single-quoted literal will work, e.g. [not '-'].

Essential Keywords

The most important keywords represent character sets, that is, any of a set of related characters.

anything, letters, digits, etc.
[character], [char], [chars], [characters] | All 256 chars (every character including NULL)
[letter], [letters] | includes é and ü, common in certain European languages.
[digit], [digits] | decimal digits 0-9
[hexdigit] | hexadecimal digits 0-9, a-f and A-F
[punctuation] | printing characters, excluding letters and digits, includes !?.,:; " ' ' / - () {} -
[symbol], [symbols] | ~@#$%^&*
  • EasyPattern's [character] or [char] will match any character including return. If you want any character except a return (or formfeed), use [paragraphChar]; that is, any character that could appear in a paragraph. Details below.
  • EasyPattern distinguishes punctuation from symbols; the sets do not overlap. For broader combinations, see [printableChar] and [typewriterChar]. For narrower focus, see [sentencePunctuation], [anyQuote], [anyBracket] and [anyDash].
  • Note that « and » are considered punctuation.
special letters
[upper], [uppercase], [uppercaseLetter] | uppercase letters
[lower], [lowercase], [lowercaseLetter] | lowercase letters
Reserved punctuation
[leftBracket] | [
[rightBracket] | ]
[leftParen], [leftParenthesis] | (
[rightParen], [rightParenthesis] | )
[comma] | ,
[singleQuote] | '
[doubleQuote], [quote] | " | (i.e. standard ASCII "straight" quotation mark)
[backwardSingleQuote] | ` |
  • EasyPattern gives special meaning to certain punctuation marks; these keywords can be used to represent the literal character.
  • Square brackets are "reserved" by EasyPattern. The other punctuation marks listed here are literal when they appear outside of brackets.
  • A literal comma and left & right brackets & parentheses may appear inside single quotes. The keywords are provided to make patterns easier to read.

Character Sets

There are many ways to create your own character sets to match exactly the characters you require.

combining character sets with "or"
[... or ...] e.g. [letter or digit], ['a' or 'b']
  • Most EasyPattern keywords refer to a set of characters from which one will match. The first use of "or" is to make a larger set -- though again, only one of the larger set will match. (Any quantity can be specified using repetition keywords, but it is still applying a quantity to a single character not to multiple characters)
  • When sets are combined with "or", parentheses are optional. (In technical terms, the character set use of "or" has very high precedence; see below)
    "[letter, letter or digit, letter]" -- matches "aaa", "xyz", "h4q", "b7f" etc.

The commas are optional here too, the "or" implicitly groups any character set keywords:
"[letter letter or digit letter]" -- same as above

Of course, it doesn't hurt to add parentheses even though they are not required.
"[letter (letter or digit) letter]" -- same as above

negation
[not ...], [non ...], [anyExcept ...] e.g. [oneOrMore non letter]
  • Instead of specifying all the characters that could occur in a match, it is often convenient to specify characters that could not occur. EasyPattern has keywords for [quotedString] and [HTMLTag], but if it didn't, they would be easy to define:
    ['<', oneOrMore not '>', '>'] « same as [HTMLTag]
    [quote, oneOrMore not quote, quote] « a simple definition for [quotedString]
  • Negation can only be applied to a single character, or a character set from which one will match. For example:
    [not letter] « fine
    [not letter or digit] « fine: [letter or digit] is a set from which one will match
    [not word] « ERROR: "word" matches multiple characters
    [not 'a'] « fine (a single character)
    [not 'whatever'] « ERROR
    [not lineChar or letter] « CAREFUL! [lineChar] is defined as [not linefeed OR verticalTab OR formfeed OR return]. You cannot combine negated and non-negated characters sets, so this pattern is equivalent to [ (not lineChar) or letter ], instead of [ not (lineChar or letter) ]
custom sets
[<...>] e.g. [<aeiou>], [<135>], [<!@#$%^&*>]
  • Keywords such as [letter] and [digit] are character sets defined internally to EasyPattern; the angle bracket notation lets you define your own characters sets. In both case, EasyPattern matches any single character in that set.
  • For single characters, [or] and a set are interchangeable, e.g. [<aei>] and ['a' or 'e' or 'i'] have the same meaning.
  • User defined sets, single character literals and EasyPattern keywords can be combined with [or]:
    [<aeiou> or <123> or '7' or symbol]

Alternatives

alternative patterns with "or"
[... or ...] e.g. ['Player' or 'EasyPattern']
  • When "or" is used to specify alternatives as part of a larger pattern, grouping parentheses are required, e.g.
    "[space, 'Player' or 'EasyPattern', space]" -- may not mean what you think!
    "[space]Player[or]EasyPattern[space]" -- may not mean what you think!
    "[(space, 'Player') or ('EasyPattern', space)]" -- that's what they mean
    "[space ('Player' or 'EasyPattern') space]" -- this might be what you wanted

Remember: as noted in the section on expressions, commas are allowed between items to make patterns easier to read; they do not affect what the pattern means.

  • If you leave out the parentheses, EasyPattern will treat everything to the left of the "or" as one implicit group and everything to the right of the "or" as a separate group. Note that visual grouping with brackets or commas is not enough; you must use parentheses. For example, all of the following will be interpreted as "[(digit, 'this') or ('that')]":
    "[digit, 'this' or 'that']" « careful; the commas may mislead
    "[digit]['this' or 'that']" « careful; the brackets may mislead
    "[digit 'this' or 'that']" « the grouping isn't clear; parentheses would help
    "[digit]this[or]that" « the grouping isn't clear; parentheses would help

As noted in the previous section, parentheses are not required when "or" is used to combine character sets.

"or" as set vs. "or" as alternative

In many cases, you don't have to worry that there are two different uses for "or"; both generally make sense in context. However, there are 2 reasons for learning the differences:

  • or as set doesn't require parentheses; the grouping is implied
  • or as set can be part of a "not" expression since it still represents one character

Repetition, Quantity

Notation: "..." is any appropriate keyword or expression, # is a number (one or more digits; the maximum varies with context).

repetition | examples | will match...
[optional ...], [zeroOrOne ...] | [digit, optional letter], [digit, zeroOrOne letter] | 2, 2a
[0+ ...], [zeroOrMore ...] | [digit, zeroOrMore letters] | 2, 2a, 2aa, 2aaa, 2aaaa...
[1+ ...], [oneOrMore ...] | [digit, oneOrMore letters] | 2a, 2aa, 2aaa, 2aaaa...
[2+ ...], [many ...], [twoOrMore ...] | [digit, many letters], [digit, twoOrMore letters] | 2aa, 2aaa, 2aaaa...
[#+ ...] | [digit, 5+ letters] | 2aaaaa, 2aaaaaa...
  • A space is not allowed, e.g. "one or more" will not be recognized
  • The words are all special cases, e.g. "threeOrMore" will not work (use "3+")
specific quantity, quantity range (where # is a number) |  | will match...
[# ...] | [5 letters] | aaaaa, bbbbb
[# to # ...] | [3 to 5 letters] | aaa, aaaa, aaaaa

shortest vs. longest match
[shortest ... ...] | match the lowest possible number of repetitions (default)
[longest ... ...] | match the highest possible number of repetitions
  • EasyPattern defaults to the SHORTEST match so the "shortest" keyword is optional.
  • When the repetition or count includes a range of values to match, EasyPattern has the choice of matching the "shortest" sequence of characters that fits the pattern, or the "longest" that fits the pattern. For example:
[shortest zeroOrOne ...] | 0 or 1 | will try to match zero occurrences
[shortest zeroOrMore ...] | 0+ | will try to match zero occurrences
[shortest oneOrMore ...] | 1+ | will try to match one occurrence
[shortest twoOrMore ...] | 2+ | will try to match two occurrences
  • In these cases, EasyPattern will only match more than the minimum if required to complete additional parts of the pattern, e.g. given "abc123" and the pattern "[shortest oneOrMore letter, digit]", EasyPattern will match "abc1", i.e. all 3 letters. However, given the same string and the pattern "[shortest oneOrMore letter]", EasyPattern will just match "a" the first letter. Given the same string and "[longest oneOrMore letter]", EasyPattern will match "abc". Note that EasyPattern always starts with the first character that matches that pattern, e.g. despite "c1" being shorter than "abc1", EasyPattern matches the latter.
  • Shortest/longest can be confusing.
  • Shortest can be quite slow, use "not" if possible

literals, groups

All of the repetition & quantity keywords can be applied to literals and groups as well as to individual keywords, e.g.
[oneOrMore 'ab'] « matches "ab", "abab", "ababab" etc.
[oneOrMore letter or digit] « matches "aaa", "456", "a45bbb" etc.
[oneOrMore not letter or digit] « matches punctuation, symbols, whitespace etc.
[oneOrMore ('alpha' or 'omega')] « matches "alphaalapha", "alphaomega" etc.
[oneOrMore (letter, digit)] « matches "r2", "r2d2", "r2d2f7b2c4" etc.

Grouping and Variable Assignment

[(...)] | A non-capturing group
[capture(...)] | Matching text is captured into 'group#' in the pattern, and into $# in the replacement. # can range from 1 to 26
[group#] | # can range from 1 to 26 e.g. [(letter)1, group1] « matches "ee", "bb", "cc" etc
[mustBeginWith(...) ...], [mustNotBeginWith(...) ...] | When a match is found, it must be/must not be preceded by what is in the brackets. The bracket contents are NOT included in the actual match. The bracket contents are limited to fixed length strings - so no '3+' etc are allowed. This must be the first part of your pattern. [mustBeginWith( 'hello' or 'goodbye' ) 'fred']
[... mustEndWith(...)], [... mustNotEndWith(...)] | When a match is found, it must be/must not be followed by what is in the brackets. The bracket contents are NOT included in the actual match. The bracket contents are limited to fixed length strings - so no '3+' etc are allowed. This must be the last part of your pattern. ['fred' mustEndWith( 'erick' or 'dy' ) ]
  • Parentheses without trailing digits form groups, e.g. to apply a quantity or repetition.
  • Commas are optional, e.g. the following patterns are equivalent:
    [( letter )1 ( digit )2]
    [( letter )1, ( digit )2]
  • By adding a number immediately after ")", you are in effect assigning the contents of the group to a variable; the variable can be referred to using "group#", described below. Note: if you need more than 26 variables, please send us an example to illustrate why!
  • Parentheses must match, i.e. ")" always ends the most recent "(", independent of number.

Comments

EasyPattern allows comments to be included in multi-line patterns using the character ';' or '#' to make the start of a comment, extending until the end of the line e.g.

[ 3 space ;look for 3 spaces
'hello' #then the keyword we want
]

Whitespace

whitespace (including items covered above)
[space], [spaces] | ASCII 32
[nonbreakingSpace] | ASCII 202
[whitespace] | [space OR tab OR cr OR lf OR verticalTab OR nonbreakingSpace]
[tab] | ASCII 9, \t
[return], [cr] | ASCII 13, \r
[linefeed], [lf] | ASCII 10, \n
[verticalTab] | ASCII 11
[formfeed] | ASCII 12, \f
[null] | ASCII 0
[CRLF] | [return, linefeed]
[newline] | [(return, linefeed) or return or linefeed]
[DOSNewline] | [return, linefeed]
[UNIXNewline] | [linefeed]
[MacNewline] | [return]
  • "not" cannot be applied to [CRLF], [newline] or [DOSNewline] since they either are or may be a character sequence rather than just a single character.
  • A space character can usually be typed directly into a pattern ([ ' ' ]) but using the keyword may make the pattern easier to understand (and modify later)
whitespace combinations
[horizontalWhitespace], [hSpace] | [space or nonbreakingSpace or tab]
[verticalWhitespace], [vSpace] | [return or linefeed or formfeed or vertical tab]
words, columns, lines & paragraphs
[wordDelimiter] | [space OR tab OR linefeed OR verticalTab OR formfeed OR return]
[wordChar] | [not wordDelimiter]
[word] | [1+ wordChar]
[columnDelimiter] | [tab OR linefeed OR formfeed OR return]
[columnChar] | [not columnDelimiter]
[column] | [1+ columnChar] Note: Use [0+ columnChar] instead if the column could be blank
[lineDelimiter] | [linefeed OR verticalTab OR formfeed OR return]
[lineChar] | [not lineDelimiter]
[line] | [1+ lineChar] Note: Use [0+ lineChar] instead if the line could be blank
[paragraphDelimiter] | [formfeed OR return]
[paragraphChar] | [not paragraphDelimiter]
[paragraph] | [1+ paragraphChar]
  • The above delimiters are characters not positions; they will "consume" the character that they match. In contrast, [TextStart] and [TextEnd] (below) are positions.
  • The above objects (word, column, line, paragraph) do not include delimiters. So, to match multiple objects, you need to include the delimiters, e.g.
    [2+ word] -- won't match anything
    [2+ (word, optional wordDelimiter)] -- correct
  • The definition for word is based strictly on whitespace so it will include punctuation, matching text such as "$27.52" and "fancy+name". Although in many cases it would be nice to exclude trailing punctuation, that pattern would fail for text like "S.M.U.". When EP's definition of a word isn't appropriate for your text, simply use the custom pattern that fits. For example, [1+ wordChar, letter or digit or symbol]" would ensure that the last char is not punctuation.
  • Word, column, line & paragraph require one or more character. If a line might be empty, use: [0+ lineChar] instead of [line].
  • Because the definitions for word, column, line & paragraph look for anything except the appropriate delimiter (rather than the leading delimiter, a series of anything else, and the trailing delimiter), they can be used to get the rest of a word, column, line & paragraph when the starting point is already in the middle. See the example scripts for details.
  • These definitions allow control characters (except the specific whitespace used as delimiters) to appear in words, columns, lines & paragraphs.
  • A column may contain the verticalTab character. (It's used by FileMaker to indicate line breaks within a field.)
  • Word, column, line & paragraph consist of multiple characters so patterns like "[not word]" don't make sense.
positions
[textStart] | matches at start of entire text
[textEnd] | matches at end of the entire text or before newline at end
[lineStart] | matches the start of a line
[lineEnd] | matches the end of a line
[wordBoundary] | matches at a word boundary
[notWordBoundary] | matches when not at a word boundary

More Keywords

combinations
[controlChar] | characters 0-31, 127 (careful: includes most whitespace)
[gremlin] | characters 0-31. The definition for [gremlin] is more cautious than in some products.
[printableChar] | [letter or digit or punctuation or symbol] (anything that prints ink on paper)
[typewriterChar] | [printableChar or space or tab or return] (excludes linefeed, vertical tab & formfeed)

punctuation subsets (these items are included in [punctuation])
[sentencePunctuation] | .,;:!?¿¡
[anyBracket], [anyBrackets] | left/right paren/bracket/brace (i.e. "bracket" in the broad sense of the term)
[anyQuote] | [doubleQuote OR singleQuote OR backwardSingleQuote]
[dash], [hyphen] | - used interchangeably. we have adopted the common notion that these terms refer to the same character
[period] | .
[caret] | ^
[pound], [hash] | #
[slash] | /
[backslash] | \
[colon] | :
[percent] | %
[star], [asterisk] | *
[ampersand] | &

real-world patterns
[HTMLTag] | <[1+ not '>']>
[HTMLStartTag] | <[not '/', 0+ not '>']> (i.e. any tag except an end tag)
[HTMLEndTag] | </[1+ not '>']>
[QuotedString] | [quote, 1+ ((backslash, quote) or not quote), quote]
[SocialSecurityNumber] | [3 digits, dash, 2 digits, dash, 4 digits]
[PhoneNumber] | Matches a US-style (xxx) xxx-xxxx number with a variety of punctuation marks. The matching text is captured into 3 successive $variables
[EmailAddress] | Matches email addresses. The name and domain parts are captured into 2 successive $variables
[IPAddress] | Matches numeric IP addresses. The matching text is captured into 4 successive $variables
[CreditCard] | Matches credit card numbers with a variety of punctuation marks. The matching text is captured into 4 successive $variables
[Hyperlink] | Matches a ftp, http, https, telnet, gopher or nntp internet url. The matching text is captured into 3 successive $variables
[DuplicateWord] | Matches a repeated word. The matching text is captured into 2 successive $variables
[PageNumber] | Matches a page number of the following forms: Page dd Page No dd Page No. dd Page Num. dd Pg Num dd Page Number dd. The matching text is captured into 3 successive $variables (Page, Number, #)
data processing patterns (in TextPipe 6.8.2 and later)
[CSVfield] | A Comma-Separated-Value field. If fields are delimited by single or double quotes, embedded newlines are allowed, as are doubled-up quotes. The quotes are returned as part of the match.
[TABfield] | A Tab-delimited field. To process multiple tab fields e.g. [ 3 or more ( TABfield tab) TABfield ]
date and time patterns
[Date] | Matches a date format DD-MM-YY or DD-MMM-YY
[AMPM] | The AM/PM part of a time
[Month] | A MonthName or a MonthNumber
[MonthNumber] | 1-12, with an optional leading zero
[MonthName] | January-December and Jan-Dec
[Day] | 1-31
[DayOfYear] | 1..366
[Year] | A 2 or 4 digit year (between 1800 and 2199)
[Hour] | A 12 or 24-hour hour, with optional leading zero
[Minute] | A 2 digit minute
[Second] | A 2 digit second

Using the real world patterns above, you can easily construct the following EasyPatterns:

HMS | [ Hour <:.-> Minute <:.-> Second ]
DMY | [ Day <-/ > Month <-/ > Year ]
MDY | [ Month <-/ > Day <-/ > Year ]
YMD | [ Year <-/ > Month <-/ > Day ]
Julian | [ Year DayOfYear ]
MY | [ Month <-/ > Year ]
MD | [ Month <-/ > Day ]
DM | [ Day <-/ > Month ]
HM | [ Hour <:. > Minute ]

Complete Patterns: Putting it all Together

A complete pattern may include many individual keywords and many expressions. How do you know which keywords go together and where one expression stops and another begins? If in doubt, just enclose every expression in parentheses. But, EasyPattern has rules for combining keywords into expressions, so parentheses aren't always required. The traditional way of expressing these rules is to list the "precedence" of various operators or terms.:

  • (...), including numbered groups
  • [or] for characters sets and single-character literal
  • [not]
  • quantity specifiers
  • character set keywords (e.g. letter, digit) and single-character literals
  • multi-character literals
  • [or] as alternative, for groups and multi-character literals

Items with high precedence don't need parentheses; they group together automatically. For example, let's build a pattern step-by-step using the "high precedence" operators:
[letter or digit] « "or" for characters set keywords
[letter or digit or '.'] « and single-character literal
[letter or digit or '.' or <!?>] « and arbitrary set
[not letter or digit or '.' or <!?>] « reverse the meaning with not
[1+ not letter or digit or '.' or <!?>] « add a quantity specifier
[1+ (not (letter or digit or '.' or <!?>))] « if you like parentheses, though the meaning is the same

Adding lower precedence terms before, after or both doesn't change the grouping, though the expression is long enough that you may find a pair of commas, brackets, or parentheses helpful. As long as you understand how EasyPattern is doing the grouping, it doesn't matter whether you choose commas, brackets or parentheses. If the parentheses are added around something that is already a group, they don't change the meaning.
[punctuation 1+ not letter or digit or '.' or <!?> symbol]
[punctuation, 1+ not letter or digit or '.' or <!?>, symbol] « same meaning but easier to read
[punctuation][1+ not letter or digit or '.' or <!?>][symbol] « same meaning
[punctuation (1+ not letter or digit or '.' or <!?>) symbol] « same meaning

Remember, commas and brackets don't change the meaning, only the look. If you put them in the middle of high precedence terms, you might confuse yourself:
[punctuation 1+ not letter][or][digit or '.' or <!?> symbol] « same meaning but HARDER to read
[punctuation 1+ not letter, or, digit or '.' or <!?> symbol] « same meaning but HARDER to read

Only parentheses change the meaning:
[(punctuation 1+ not letter) or (digit or '.' or <!?> symbol)] « different meaning

Note that "or" for character sets and "or" as alternative have opposite precedence. See Character Sets and Alternatives (above) for details & examples.

EasyPattern vs. perl regex or grep

At its core, EasyPattern uses "regular expression" technology that is similar to the "regex" or "grep" tools that originated on UNIX. EasyPattern's primary benefit is that the patterns are much easier to read and write.

For those who have some experience with regex, here are a few specific differences:

  • Quantity is specified as a prefix rather than a suffix. We believe prefix notation is much more natural.
    e.g. "[1+ digit]" rather than "[0-9]+"
  • Parentheses groups are not automatically numbered. Drawback (to some): you have to include a number if you want to refer to that matched portion. Benefits: the numbers don't change when you add other parentheses, you can number only the groups that you want to use (the parentheses that are there just for logical grouping don't get numbered).
  • No backslashes are required to "escape" special characters (instead, EasyPattern provides keywords such as rightBracket). Benefit: Other pattern languages already use backslash as an escape character so extra backslashes make patterns even more difficult to read.
  • EasyPattern includes keywords for many character sets that require a custom bracketed set in regex, e.g. punctuation, whitespace, paragraph, column, etc.
  • EasyPattern keywords generally include Macintosh-specific characters, e.g. [letter] includes letters with umlauts and other diacritical marks
  • EasyPattern can combine character sets with "or" (as well as use "or" for alternatives).
  • EP's [character] or [char] will match any character; the "equivalent" in some products will match anything except carriage return. If you want any character except a return (or formfeed), use [paragraphChar]; that is, any character that could appear in a paragraph. Of course, [not return] works too.


Advanced Topics

 Advanced Topics

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up
Automation Examples
Command Line Options
Optimizing Performance
Optimizing Performance
Standard ASCII codes
Decimal to Hex Conversion Chart
Null characters - Clipboard Limitations
Registry entries
File Size Limitations
Unprintable Characters
Task Bar Menu
Logging
Exit codes
Data Mining
COM Interface
Troubleshooting command-line automation
Unattended install

Optimizing performance, temporary files
Automation Examples
 Command Line Options
 Troubleshooting your filters
Troubleshooting command line automation
COM interface for scripting
Defining custom patterns
Unprintable Characters
Null characters - Clipboard Limitations
 File Size Limitations imposed by the Operating System, or by some filters
 Task bar menu
Logging
Exit codes
Registry entries
Data mining
Standard ASCII codes
 Decimal to Hex Conversion Chart
System requirements


Task Bar Menu

 Task Bar Menu

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

The task bar shows a small TextPipe icon while TextPipe is running.

By right clicking on the icon, you get a menu of options. The options below the line are filters that you can run.

Typically the filters process text that is on the clipboard. This is so that you can copy text from your editor or Word Processor to the clipboard, run a TextPipe filter via the taskbar icon, and then paste the converted text back into your application.

You can also process files e.g. copy a file to the clipboard (e.g. to copy your favorite mail signature to the clipboard ready for pasting).

To add your own filters to the list shown, just copy or move them to the folder <TextPipe Install Folder>\TaskMenu. for a default installation this folder is C:\Program Files\TextPipe\TaskMenu.

Go

Clicking Go causes the current filter to be run (rather than a filter on the taskbar menu)

Clipboard

Clicking Go causes the current filter to be run (rather than a filter on the taskbar menu) with the input and output forced to the clipboard (much the same as the Clipboard button in the main window).

Force clipboard

This option causes any filters clicked on from the list below it to be run with the input and output forced to the clipboard (much the same as the Clipboard button in the main window), regardless of how they may be defined.

Clear clipboard

Clears the contents of the clipboard

Refresh list

Causes TextPipe to re-read the list of files in the TaskMenu folder.


Logging

 Logging

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

The filter options pane can be reached by selecting the topmost node in the filter tree. In the registered version it allows you to set logging profiles. Logging allows you to track when a filter was run, who ran it, how long it ran for, which files it processed or skipped, the input size and output size of each file and the changes made to it by each filter. It also keeps track of any errors that occurred during a filter job.

There are three levels of log entry - Info, Warning and Error. Info lines are reported in the Results tab of the Status Window, and Warnings and Errors are reported in the Errors tab. All three are written to the log file. Logged errors modify TextPipe's exit code.

Logging enabled

When checked, this allows log information to be sent to the log file.

Log filename

The name of the file to log to. In the evaluation version this is forced to the same folder than textpipe.exe is in. TextPipe will force the creation of folders if necessary.

Show

Opens the log file in NotePad.

Delete

Deletes the log file after prompting for confirmation.

Append to file

Adds to an existing log file if one exists

Overwrite file

Clears any existing log file before adding log data

Don't log to file if line count is less than...

No lines are written to the log file unless this threshold is exceeded. Any lines are stored in the results screen and can be transferred to the log file if desired. A value of zero forces all lines to be written out.

This is useful for preventing logging for small jobs, and keeps the size of the log file small.

See also

Command line parameters


Automation Examples

 Automation Examples

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Command line automation can be initiated in two basic ways (see here for COM automation):

  1. via a batch file
  2. via a Desktop or Start Menu shortcut (create a TextPipe shortcut, then change the Target property under the Short Cut tab)

Either way, you need to be careful to quote any long filenames e.g.

"c:\program files\TextPipe\TextPipe.exe" /f=filter_file.fll "c:\my documents\*.*" /g /q

You can use the command line automation helper to give you a variety of examples of automating the current filter.

This simple line starts TextPipe, opens the filter called filter_file.fll, then runs it across all files in c:\my documents\*.*. The /g option tells TextPipe to "start now", and the /q option tells TextPipe to "quit when done". Omitting either of these options or putting them in the wrong order will prevent automation from working.

A set of filtering options can also be specified using the command line (rather than using a filter file). Not all filtering options can be specified on the command line. This limitation can be avoided completely by saving a filter (.fll file) with the appropriate filtering options, and then using the /F=<filter file> option to open it from the command line.

MS-DOS Prompt START command

You can use the MS-DOS Prompt's START command to force the command shell to wait for TextPipe to finish before it continues with the next command. This can be very useful if you run a series of two or more TextPipe filters in sequence (also referred to as 'chaining' filters together). The full command line options for the Start command are:

C:\>start /?
Runs a Windows program or an MS-DOS program.

START [options] program [arg...]
START [options] document.ext

/m[inimized] Run the new program minimized (in the background).
/max[imized] Run the new program maximized (in the foreground).
/r[estored] Run the new program restored (in the foreground). [default]
/w[ait] Does not return until the other program exits.

So to run multiple TextPipe filters in a row, use a sequence like the following:

start /w "c:\program files\TextPipe\TextPipe.exe" /f=filter1.fll "c:\my documents\*.*" /g /q
start /w "c:\program files\TextPipe\TextPipe.exe" /f=filter2.fll "c:\my documents\*.*" /g /q
...


Data Mining

 Data Mining

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

What is Data Mining?

Data mining or text mining is where a source of information is processed to extract information. Examples include:

  • Process a web site to extract product catalog and cost information. This can then be used to compare prices between different suppliers.
  • Process a web site to extract email addresses or web URLs.
  • Harvest the data on a web site for your own purposes.

Extracted data is designed to be easily loaded into a database for further analysis.

Click here for our white papers on Web Site Data Mining, Report Mining and more

How can TextPipe help?

TextPipe can be used to generate an extract from any text data source, including web sites. TextPipe can also be used to perform data cleansing or any additional processing e.g.

  • add a header record (e.g. provide column titles for .CSV files)
  • remove unwanted data
  • replace specific text
  • convert line feeds to DOS/Unix/Mac
  • expand tabs
  • fix capitalization
  • convert from EBCDIC to ASCII
  • remove multiple whitespace
  • remove columns, lines or fields
  • remove duplicate records
  • sort
  • extract email addresses from specific fields
  • discard records matching a pattern
  • and much more

Click here for more information about TextPipe

How can Offline Explorer help?

MetaProducts Offline Explorer (OE) can download your favorite Web, FTP and HTTPS sites (up to 100 simultaneously) for later offline viewing, editing or browsing (OE has a built-in internal browser). Offline Explorer allows you to selectively (include or exclude) individual servers, directories, and files using only keywords. OE is known for its excellent user interface and as one of the fastest known Web site downloaders available on the market today. Support of industry-standard technologies, like FTP, HTTPS, different proxy servers, Macromedia Flash, Cookies, XML, a built-in internal HTTP server that allows you to share downloaded files over an Intranet, makes Offline Explorer the leader in offline browsing.

For data mining applications, Offline Explorer can run TextPipe when a web site has completed downloading. OE knows which file types contain text, so it tells TextPipe to only process those files.

TextPipe can also be used to reduce the size of downloaded web sites:

  • convert line feeds from DOS/Unix/Mac to None
  • remove multiple whitespace
  • remove whitespace at the start and end of each line
  • remove blank lines
  • remove binary characters

Click here for more information about Offline Explorer

Contact Us Support Community Tutorials and User Guides (online)
Copyright © 1999-2005 Crystal Software Australia . All rights reserved.


COM Interface

 COM Interface

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up
COM examples

You can script TextPipe's powerful text-processing functions from any external programming language using TextPipe's COM interface. You can also use the File Menu\Export functions to generate VBScript or JScript code to create and run a given filter. A sample script is included in the VBScript\sample.vbs file. You can also see our Web page sample.

Notes:

  1. Before adding filters to the tree, you MUST call startFilters. When you've finished adding filters, you MUST call endFilters. Together with startSubfilters and endSubfilters these control the depth that the filters are added in the tree.
  2. To create sub filters you MUST surround them with startSubfilters and endSubfilters commands.
  3. Values that are NOT indicated as allowed are not supported and may cause errors.

The same interface is supported by both TextPipe.Application (the interactive application) and TextPipeEngine.Application (the ultra-high speed component, found in the separately purchased TextPipe Engine). The icon XX is used to indicate methods or functions that do nothing in TextPipeEngine.Application - most of these functions relate to GUI functions.

TextPipe.Application and TextPipeEngine.Application

Function/Method/Property | Description
newWindow() : TextPipe.FilterWindow | Returns a new TextPipe.FilterWindow object that holds a new TextPipe filter window
minimize() | Minimizes TextPipe's main window
restore() | Restores TextPipe window to its previous size
hide() | Hides TextPipe's window
show() | Shows TextPipe's window
version() : string | Returns the TextPipe version number string
findWindow( filename : string ) : TextPipe.FilterWindow | Searches for an existing window that already has the given filter filename loaded, and returns it if found. If not found, it opens a new window, loads the given filter into it and returns it.
calledFrom( applicationName : string ) | Sets the calling application name. When set, the File\Exit menu item changes to 'Exit and return to applicationName', and when the user tries to close TextPipe it simply hides itself. This is very handy when calling TextPipe from a third party application.
close() | Attempts to close TextPipe, prompting the user to save any modified filters.
quit() | Closes TextPipe WITHOUT prompting the user to save any modified filters. This function is the only way to close TextPipe after it has been made visible (even if TextPipe is made invisible again)
property visible : boolean | True if the main form is visible, False if it is not. When called from COM, TextPipe's window is initially invisible. As soon as TextPipe is made visible, it locks itself in memory to prevent itself being closed by COM. The only way to close it is to call the Quit function (making it invisible does not change this)
property activeFilter : TextPipe.FilterWindow | Returns the top-most filter window. If none, returns nil.
property installationFolder : string | Returns the folder where textpipe.exe file is located, with a trailing backslash. This can be used to find the correct local path to load a pre-defined filter from.
property Application : TextPipe.Application | Returns the Application object
property Parent : TextPipe.Application | Returns the Application object
property currentDirectory : string | Gets or sets the current directory. This is used by DataPipe to force TextPipe to save filters in DataPipe's filter directory rather than in a directory assigned (randomly) by Windows.

TextPipe.FilterWindow

Open and closing files
openFilter( filename : string ) | Opens a filter with the specified filename
saveFilter( filename : string ) | Saves the current filter with the given filename
close() | Closes the filter window
closeWithoutSave() | Same as close but closes the window without prompting even if the filter has been changed
property filterChanged : boolean | Read/Write. True if the current filter has been saved, False if it has been changed
property scratchPad : string | Read/Write. The scratch pad text
property trialInput : string | Read/Write. The trial input text
property comments : string | Read/Write. The comments tab text.
property isRunning : boolean | Read Only. True if the filter is currently running.
Filters
Before any filters are added or any Logging, Input or Output settings are changed, you must call

startFilters()

first, and then call

endFilters()

when you have finished adding or changing. These function turn off the GUI, making adding filters much faster. They also prevent the GUI settings from overwriting the settings you make.

To create sub filters, you must surround each set of sub filters like this:

startSubfilters()

...define filters here...

endSubfilters()
clearAllFilters() | Removes all filters from the filter list
moveCursorToEnd() | Moves the cursor in the treeview to the end of the list. By default, when a filter is loaded the cursor is on the Input Filter. If you want to add additional filters you generally will want to move the cursor to the end of the filter list first.
makeSubfilterOf( parent : treenode, child : treenode ) This function has been superseded by the startSubfilters and endSubfilters commands.

Moves the child filter so that it becomes the last child sub filters of the parent filter. You must determine if this action is legal or not yourself - TextPipe does no error checking.

addSelectionFilter( type : integer, locateMethod : integer, param1 : integer, param2 : integer, moveTo : integer, processIndividually : boolean, excludeDelimiter : boolean, excludeQuotes : boolean; delimiter : integer; customDelimiter : string; hasHeader : boolean ) : treenode Adds a filter from the Restriction menu. Note that Restriction filters almost always require sub filters in order to have any effect.

Type - the type of filter to add

0 Delete column
1 Restrict lines
2 Restrict columns
3 Restrict to bytes
4 Restrict to delimited fields (CSV, Tab, Pipe etc)
5 Restrict tab-delimited fields (deprecated, use 4 instead)
6 Remove lines
7 Remove delimited fields (CSV, Tab, Pipe etc)
8 Remove Tab-delimited fields (deprecated, use 7 instead)
9 Move columns
10 Move delimited fields (CSV, Tab, Pipe etc)
11 Move Tab-delimited fields (deprecated, use 10 instead)
12 Copy columns
13 Copy delimited fields (CSV, Tab, Pipe etc)
14 Copy Tab-delimited columns (deprecated, use 13 instead)
15 Restrict to delimited fields with column header (deprecated, use 4 instead)
16 Restrict to tab fields with column header (deprecated, use 4 instead)
17 Remove Byte Range
18 Delete pipe-delimited fields (deprecated, use 7 instead)
19 Restrict to pipe-delimited fields (deprecated, use 4 instead)
20 Restrict to pipe-delimited fields with header (deprecated, use 4 instead)
21 Move pipe-delimited fields (deprecated, use 10 instead)
22 copy pipe-delimited fields (deprecated, use 13 instead)

Locate Method - how to determine which areas to affect

0 Restrict %d .. %d
1 Restrict %1:d starting at %0:d
2 Restrict %1:d starting at END - %0:d
3 Restrict %d .. END - %d
4 Restrict END - %d .. END - %d

param1, param2 - the required integer values for the locate method

moveTo : integer - where to move or copy the columns or fields to. Default 1.

processIndividually - whether or not to apply sub filters to each CSV or Tab field individually, or to the fields as one string value. Default false.

excludeDelimiter - whether or not to include the comma or Tab field delimiter when passing the field to the sub filter. Default true.

excludeQuotes - whether or not to include the CSV quotes that may surround the field when passing the field to the sub filter. Default true.

delimiter - (optional) the index of the standard delimiter to use, or 6 for custom, default 0 for CSV

customDelimiter - (optional) the custom delimiter to use, default blank

hasHeader - (optional) true if the file's first row is a header row, default false.

addGrepFilter( type : integer, pattern : string, includeLineNumbers : boolean, includeFilename : boolean, ignoreCase : boolean, countMatches : boolean, patternType : integer ) : treenode Adds a Grep type line based filter.

type

0 Restrict lines matching
1 Restrict lines NOT matching
2 Extract
3 Retain matching lines
4 Retain non-matching lines
5 Remove matching lines
6 Remove non-matching lines

pattern - the pattern to match

includeLineNumbers - true to include the line number where the pattern was found, default False

includeFilename - true to include the filename where the pattern was found, default False

ignoreCase - ignore case when matching the pattern, default True

countMatches - true to only output a count of the number of matches, default False

patternType

0 Perl pattern
1 Egrep pattern
2 Brief pattern
3 MS Word pattern
4 EasyPattern
addSplitFilter( type : integer, splitSize : integer, splitChar : string, splitCharPos : integer, splitCharCount : integer, splitLines : integer, splitFilename : string, firstfileNumber : integer ) : treenode Adds a split type filter

type

0 Split at a given size
1 Split at a given character
2 Split at a given number of lines

splitSize - the size file to split at

splitChar - the character to split at

splitCharPos -

0 Split before the character (it goes into the next file)
1 Split after the character (it remains in the first file)
2 Split on top of the character (remove it)

SplitCharCount - the number of times to see SplitChar before splitting

SplitLines - (optional) split after a given number of lines, default 60

SplitFilename - (optional) the name to give to each output split file, default %f.%e.%3.3d

firstFileNumber - (optional) the number of the first file, default 0

addMergeFilter( type : integer, filename : string ) : treenode Adds a merge type filter

type

0 Merge into filename
1 Retain lines found in filename
2 Remove lines found in filename
3 Link filter filename

filename - the filename to use

addHeadTailFilter( includeOrExclude : boolean, linesOrBytes : integer, startOrEnd : boolean, count : boolean, ) : treeNode Adds a Head/Tail type filter

includeOrExclude - whether to include or exclude the text, False for Include, True for Exclude

linesOrBytes - measure in terms of lines or bytes, False for Lines, True for Bytes

startOrEnd - measure from the start or end of the file, False for Start, True for End

Count - the count of lines or bytes to include or exclude

addRunFilter( commandLine : string, inputFilename : string, outputFilename : string ) : treeNode Adds a Run External Program filter

commandLine - the command line of the program to run. Should include double quotes around long filenames, and also usually includes the input and output filename.

inputFilename - the filename that TextPipe should read from after the External Program writes to it.

outputFilename - the filename that TextPipe should write to for the External Program to read in.

addSortFilter( type : integer, reverse : boolean, removeDuplicates : boolean, startColumn : integer, length : integer ) : treenode Adds a sort type filter

type - the sort type

0 ANSI sort (case insensitive)
1 ANSI sort (case sensitive)
2 ASCII sort (case insensitive)
3 ASCII sort (case sensitive)
4 Numeric sort
5 Sort by length of line
6 Sort by date and time
7 Sort by date
8 Sort by time

reverse - whether to reverse the normal sort order, when set to True the order is Descending, default false

removeDuplicates - whether to remove duplicate values or not, default true

startColumn - the column to start comparisons from

length - the length of the comparison

addDuplicatesFilter( type : integer, ignoreCase : boolean, startColumn : integer, length : integer ) : treenode Adds a Show or Remove Duplicates filter

type -

0 Remove duplicate lines
1 Show duplicate lines

ignoreCase - whether to ignore case during comparisons or not

startColumn - the starting column for the comparison

length - the length of the comparison

addReplaceFilter( searchStr : string, replaceStr : string, type : integer, matchCase : boolean, wholeWord : boolean, caseSensReplace : boolean, promptOnReplace : boolean, extract : boolean, firstOnly : boolean, skipPromptIdentical : boolean ) : treenode Adds a search and replace (find and replace) filter. See also the setPerl, setEditDistance and setBufferSize functions (below) for setting additional parameters.

searchStr - the string to search for

replaceStr - the string to replace it with

type - the find type:

0 Replace
1 Pattern (old style)
2 Sounds like
3 Edit distance
4 Perl pattern
5 Brief pattern
6 Word pattern
7 EasyPattern

matchCase - Matches case when set to True, ignores case when set to false (default)

wholeWord - Matches whole words only when set to True, default false

caseSensReplace - Replaces with matching case when set to True, default false

promptOnReplace - Prompts before replacing when set to True, default false

extract - when True, all non-matching text is discarded, default false

firstOnly - when True, only replace the first occurrence, default False

skipPromptIdentical - when True, don't bother prompting if the replacement text is identical to the original.

addReplaceListFilter( filename : string, type : integer, matchCase : boolean, wholeWord : boolean, caseSensReplace : boolean, promptOnReplace : boolean, firstOnly : boolean, skipPromptIdentical : boolean ) : treenode Adds a search and replace list filter. See also the setPerl, setEditDistance and setBufferSize functions (below) for setting additional parameters.

filename - the file to load search/replace pairs from

Other options - as above.
setEditDistance( threshold : integer ) : boolean | Sets the edit distance threshold for the immediately preceding search/replace filter. Default 2.
setBufferSize( bufferSize : integer ) : boolean | Sets the buffer size for the immediately preceding search/replace filter. Default 4096.
setPerl( bufferSize : integer, greedy : boolean, allowComments : boolean, dotMatchesNewLines : boolean ) : boolean Sets the perl matching options for the immediately preceding search/replace filter.

bufferSize - the maximum buffer size to use for matches. Any match must fit into this buffer, so if you want to match larger pieces of text, increase the size of this buffer to suit. Default 4096.

greedy - if the pattern finds the longest match (greedy) or the shortest match. Default false.

allowComments - allow comments in the perl pattern. Default false.

dotMatchesNewLines - allow the '.' operator to match all characters, including new lines. Default true.

addNumberFilter( type : integer, value : integer ) : treenode Adds a Number-type filter.

type - the type of filter to add

0 Convert Tabs to Spaces
1 Convert Spaces to Tabs
2 Word wrap (column width %d)
3 Pad to width of %d
4 Centre in width of %d
5 Right justify in width of %d
6 Restrict CSV field %d
7 Restrict tab-delimited field %d
8 Truncate to width %d
9 Force to width %d
10 Repeat file %d times
11 Restrict to blocks of length
12 Expand packed decimal (with implied decimals)
13 Expand zoned decimal (with implied decimals)

Value - the numeric value to use

addInsertFilter( insertPos : integer, insertString : string, type : integer ) : treenode Adds an insert type filter.

insertPos - the position to insert the string

insertString - the string to insert

type - (optional) the type of filter to add-

0 Insert column (default)
1 Insert bytes
addDatabaseFilter( connectionStr : string, mode : integer, fieldDelimiter : string, textQualifier : string, generateHeader : boolean, insertTable : string, timeout : integer ) : treenode Adds a database-type filter.

connectionStr - the database connection string.

mode

0 Delimited output
1 Fixed width
2 XML
3 Insert script

fieldDelimiter - the string to use between columns

textQualifier - the string to use around string column values

generateHeader - generates header information when True

insertTable - the name of the insert table

timeout - SQL command timeout in seconds, default 30

addMapFilter( filename : string ) : treenode Adds a single byte map-type filter from a file

filename - the filename containing the map

See also prepareMapFilter

prepareMapFilter() : treenode

setMap( characterIndex : integer, outputString : string )
setMap( characterIndex : integer, outputString : string )
setMap( characterIndex : integer, outputString : string )
...

Adds a single byte map-type filter and allows it to be configured by the following statements

See also addMapFilter

prepareUnicodeMapFilter( unenteredValue : integer, replaceWith : string ) : treenode

setUnicodeMap2( startRange, endRange, outputString : string )
setUnicodeMap( characterIndex : integer, outputString : string )
setUnicodeMap( characterIndex : integer, outputString : string )
setUnicodeMap( characterIndex : integer, outputString : string )
...

Adds a Unicode map-type filter and allows it to be configured by the following setUnicodeMap or setUnicodeMap2 statements.

unenteredValue:

  • 0 - Pass through unchanged
  • 1 - Pass to subfilter
  • 2 - Replace with value (use next option to specify it)
  • 3 - Remove

Replace with:

The string to replace it with, if the Replace Type is 2.
addEOLfilter( inputLineFeeds : integer, fixedLength : integer, outputLineFeeds : integer, newLineFeed : string, removeBadEOLs : boolean ) : treenode Adds an EOL (end of line) conversion filter

inputLineFeeds - the type of incoming line feeds

0 Unix
1 Mac
2 DOS
3 Auto
4 Fixed (use the fixedLength parameter to specify the length)

fixedLength - the length of fixedLength input files when option #4 is chosen above.

outputLineFeeds - the type of outgoing line feeds

0 Unix
1 Mac
2 DOS
3 None
4 String (use the newLineFeed parameter to specify the new string)

newLineFeed - the new line feed string on output when option #4 is chosen.

removeBadEOLs - whether to remove bad EOLs or not

addSimpleFilter( type : integer ) : treenode Adds a simple filter type, requiring no special parameters.

type - the type of filter to add.

1 Convert ASCII to EBCDIC
2 Convert EBCDIC to ASCII
3 Convert ANSI to OEM
4 Convert OEM to ANSI
5 Convert to UPPERCASE
6 Convert to lowercase
7 Convert to Title Case
8 Convert to Sentence Case
9 Convert to tOGGLE cASE
10 Remove blank lines
11 Remove blanks from End of Line
12 Remove blanks from Start of Line
13 Remove binary characters
14 Remove ANSI codes
15 Convert IBM drawing characters
16 Remove HTML and SGML
17 Remove backspaces
18 Resolve backspaces
19 Remove multiple whitespace
20 UUEncode
21 Hex Encode
22 Hex Decode
23 MIME Encode (Base 64)
24 MIME Decode (Base 64)
25 MIME Encode (Quoted printable)
26 MIME Decode (Quoted printable)
27 UUDecode
28 Extract email addresses
29 Unscramble (ROT13)
30 Hex dump
32 XXEncode
33 XXDecode
34 Reverse line order
35 Remove email headers
36 Decimal dump
37 HTTP Encode
38 HTTP Decode
39 Randomize lines
40 Create word list
41 Reverse each line
42 Convert to RanDOm case
43 Extract URLs
44 ANSI to Unicode
45 Unicode to ANSI
46 Display debug window
47 Word concordance
48 Delete all
49 Restrict to each line in turn
50 Convert CSV to Tab-delimited
51 Convert CSV to XML' )
52 Convert Tab-delimited to CSV
53 Convert Tab-delimited to XML
54 Convert CSV (with column headers) to XML
55 Convert Tab-delimited (with column headers) to XML
56 Convert CSV (with column headers) to Tab-delimited
57 Convert Tab-delimited (with column headers) to CSV
58 Restrict to file name
59 Convert Word documents to text
60 Swap UTF-16 word order
61 Swap UTF-32 word order
62 Remove BOM (Byte Order Mark)
63 Make Big Endian
64 Make Little Endian
65 Compress to Packed Decimal
66 Compress to Zoned Decimal
addStringFilter( type : integer, str : string ) : treenode

addStringFilterFile( type : integer, filename : string ) : treenode

Adds a string-type filter.

type - the type of filter to add

0 Add left margin
1 Add header
2 Add footer
3 Add right margin
4 Remove lines
5 Retain lines
6 Remove lines matching perl pattern
7 Retain lines matching perl pattern
8 Add text side by side
9 Add repeating text side by side
10 Not Used
11 Not Used

str - the string to use

filename - the filename to use

addLineNumberFilter( startNumber : integer, increment : integer, skipBlankIncrement : boolean, dontNumberBlank : boolean, numberFormat : string ) : treenode Adds a Line Number filter

startNumber - the starting line number

increment - the amount to add for each new line number

skipBlankIncrement - don't increase the line number for blank lines

dontNumberBlank - don't put a line number on blank lines

numberFormat - the format to use for the line number

addScriptFilter( language : string, script : string, timeout : integer ) : treenode Adds an ActiveX script filter

language - the language of the script

script - the code

timeout - the command timeout in seconds, default 30

addCommentFilter( comment : string ) : treenode Adds a comment type filter

comment - the comment.

addMathsFilter( operation : integer; operand : integer ) : treenode Adds a maths type filter

operation - the operation to perform

0 +
1 -
2 *
3 div (the remainder is ignored)
4 mod (the remainder after division)
5 xor
6 and
7 or
8 not
9 shift left (0 inserted)
10 shift right (0 inserted)
11 rotate left
12 rotate right

operand - the operand to use

addXMLFilter( findType : integer; tag : string; attribute : string; endTag : string; includeText : boolean; includeQuotes : boolean; matchCase : boolean ) : treenode Adds an HTML/XML filter

findType - the operation to perform

0 restrict to an element
1 restrict to an attribute
2 restrict to between tags

tag - the element or start tag to find

attribute - the attribute to find

endTag - the endTag to find

includeText - whether to include the find string in the restriction result (default false)

includeQuotes - whether to include surrounding quotes in the attribute result or not (default false)

matchCase - match case exactly or not (default false)

addUnicodeConversionFilter( toUnicode : boolean; conversion : integer) : treenode Adds a Unicode conversion filter

toUnicode - true to convert TO Unicode, false to convert FROM Unicode

conversion - the conversion to perform, from 1 to 144 or so. You can determine the correct value by exporting a filter that contains the conversion filter you require.

Output Filter
Note - Before changing these settings you must call startFilters first, and call endFilters when you have finished.
property outputMode : integer Read/Write. Sets the output mode:

0 Output to clipboard (all files are joined together)
1 Output to files
2 Output merged (joined) to a single file
property outputTestMode : integer Read/Write. Sets the output filter test mode.

0 Normal mode
1 test mode
2 Backup mode.

property outputFolder : string | Read/Write. Sets the output filter folder.
property outputRetainDate : boolean | Read/Write. Sets the output filter retain date setting. True turns on the retain date setting, default False.
property outputMergeFile : string | Read/Write. Sets the output filter to merge data to a single file, using the given filename.
property outputExtension : string | Read/Write. Sets the output filter extension.
property outputAppend : boolean | Read/Write. Sets append mode when True, default False.
property outputOnlyOutputChangedFiles : boolean | Read/Write. Only outputs files that have actually changed, default True.
property outputKeepStructure : boolean | Read/Write. When used in conjunction with outputFolder, determines if the original folder structure is retained, default False.
addSecondaryOutputFilter( mode : integer, testMode : integer, retainDate : boolean, append : boolean, onlyOutputChangedFiles : boolean, folder : string, extension : string, keepStructure : boolean, mergeFilename : string ) : treenode | Adds a secondary output filter, usually used for splitting some lines or records to a new file. All parameters are as above for a normal Output filter.
Input Filter
Note - Before changing these settings you must call startFilters first, and call endFilters when you have finished.
property InputMode : integer Read/Write. Sets the input mode:

0 Input from clipboard
1 Input from files (default). You can add files using the addFile function.
2 Input from Trial Run area. You can set the Trial Run text using the trialInput property
3 Input from random file generator. You can set the

property inputBinaryFiles : integer Read/Write. Controls how binary files are handled

value (default 0)

0 Binary files are processed
1 Binary files are skipped
2 Binary files are confirmed before processing

property InputBinarySampleSize : integer | Read/Write. Default 100. Controls the binary sample size for Input files
property InputPromptOnEach : boolean | Read/Write. Default False. Controls whether to prompt before processing each file.
property InputPromptOnReadOnly : boolean | Read/Write. Default False. Controls whether to prompt before processing each read only file.
property InputDeleteFiles : boolean | Read/Write. Default False. Controls if input files are deleted after processing. Use with caution!
property inputRandom : integer | Read/Write. Sets the input to come from a randomly generated file of size size, default 1024
Logging functions
Note - Before changing these settings you must call startFilters first, and call endFilters when you have finished.
property logEnabled : boolean | Read/Write. When True, logging is on, default False
property logFilename : string | Read/Write. The file to log data to
property logAppend : boolean | Read/Write. When True, logging appends to the existing file
property logThreshold : integer | Read/Write. When more than threshold lines are present, all get written to the log file (this is for interactive mode only, for non-interactive mode ALL lines get written to the log file)
Running Filters
execute() : boolean Runs the current filter (same as the 'Go' button). This function does not return until the thread completes.

Returns true on success, False on failure (with error message in errorText)

executeClipboard() : boolean Runs the current filter with input from and output to the Clipboard (same as the 'Clipboard' button). This function does not return until the thread completes.

Returns true on success, False on failure (with error message in errorText)

processString( inputText : string ) : string Processes the string inputText and returns the result string, just as though the entire string was in a file.

If an error occurs, errorText will be set to the error message, or blank if there is no error.

compileFilter() : string Compiles the filter into an internal form and saves it so that the compiling doesn't have to be repeated each time text is processed. This function is ONLY used in conjunction with processString(s)WithCompiledFilter - it does not speed up any other process. Any changes made to the filter list after the filter was compiled will not be reflected. Also, the normal Input and Output filters are ignored - a string input and output filter is used instead.

Returns a blank string on success, error message on failure. errorText is also cleared if there is no error, or set if there is an error.

processStringWithCompiledFilter( inputText : string ) : string | Processes the string inputText using the pre-compiled filter. If the filter list has not been compiled with compileFilter this function will return an empty string. This function calls startJob and endJob at the beginning and end of processing. Note the singular - String
compiledFilterStartJob()
processStringsWithCompiledFilter( inputText : string ) : string
...
(usually called multiple times)
...
compiledFilterEndJob() 		Processes the string inputText using the pre-compiled filter. If the filter list has not been compiled with compileFilter this function will return an empty string. This function DOES NOT call startJob and endJob at the beginning and end of processing - they must be called separately. This function is normally called a number of times. Note the plural - Strings

compiledFilterStartJob() - this function prepares a compiled filter for processing.

compiledFilterEndJob() - this function completes a compiled filter's processing.

isRunning() : boolean | Returns true if a processing thread is running
property errorText : string | Returns the text of the last error, or blank if there was no last error.
Adding Files
clearAllFiles() | Clears all files from the file list
addFile( fileSpec : string, subfolders : integer, action : integer ) Adds a file to the file list.

fileSpec - the filename, folder or wildcard

subfolders - how many subfolders to include (default 255)

0 No subfolders
1..9 1..9 Subfolder(s)
255 All subfolders

action - the action column (default 1)

1 Include the files
2 Exclude the files
3 Ignore the files
addFileList( filename : string ) Adds a list of files to the file list.

filename - the list of files is contained in this file.


Command Line Options

 Command Line Options

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

TextPipe can be run from a DOS shell or from a shortcut with parameters to make it run completely automatically. This is very useful for performing conversions automatically or on machine start up (by making the short cut an entry in the Start Up folder). You can also use the command line wizard to provide examples. If you are having difficulties, please see the Troubleshooting command-line automation page. TextPipe sets an exit code on completion.

Note:

  1. Most commands change settings or add filters to the current filter. If you load a filter with /F=, the /F= command must appear FIRST, and any commands that modify it MUST come AFTER, otherwise the loaded filter's settings will override prior commands.
  2. Any commands with embedded spaces must be surrounded with double quotes. You must surround the entire command/setting combination e.g. "/f=c:\Program Files\TextPipe\My Filters\test.fll" NOT /f="c:\Program Files\TextPipe\My Filters\test.fll"
  3. Windows limits the command line length to about 260 characters. You can overcome this limitation by using the /Z=<filename> command to load command line parameters stored in a file. Note that you MUST remove any double quotes from around commands with embedded spaces when you put them in the command file.
  4. If you want to add filters to a command-line loaded filter at run time, you must ensure that the very last filter (prior to the output filter) is not a sub filter, otherwise all added filters will be added as sub filters.
  5. Not all filters or options can be set via the command line. To workaround this limitation, save a filter with your desired settings (e.g. an output base directory of 'c:\temp'), and then open the filter as the first item on the command line (using /f=<filter name>). If you use /F= a second or third time, these filters will be appended to the existing filter.
  6. If you want TextPipe to start minimized, use /MINIMIZED as the very FIRST command line parameter, or make sure that it is on the command line and NOT in the command file.

You can automatically generate command line text from your filter using the File Menu\Export\Export command line to clipboard feature.

Conventions used below:

<> surrounds text that is to be entered. Do NOT enter the '<' and '>' characters

Alternatives are shown in square brackets '[' and ']' with a pipe '|' character between them. Do NOT enter the square brackets or the pipe character - just type the single alternative you require.

Usage:

"c:\program files\TextPipe\textpipe.exe" [/MINIMIZED|/MAXIMIZED] ["/F=my filter file.fll"] [commands] [wildcards] [/G [/Q]]

Where [commands] is zero or more of:

<wildcard>
 A wildcard or filename to add to the list of files to process e.g.

"c:\Documents and Settings\Simon\Desktop\*.txt;*.ans"

/{=<string>
 Adds a comment. See comment filters.
/A[S|F|T|E]=<string>
 Adds a string:
  • ‘S’ Adds a left margin to every line (like email quotes etc)
  • ‘F’ Adds a right margin to every line
  • ‘T’ Adds a header to the start of every file (e.g. banner advertising)
  • ‘E’ Adds a footer to the end of every file (e.g. copyright notices)
  • '1' Remove lines from list
  • '2' Retain lines from list
  • '3' Remove lines matching perl pattern
  • '4' Retain lines matching perl pattern
  • '5' Add text side by side
  • '6' Add repeating text side by side

/AL=start,increment
 Add line numbers, starting from start and with the given increment e.g.

/AL=500,50

See Line number filters.

/B<C|P|S>
 Skip binary files. Default: Skip. Use
  • 'P' for Process all binary files
  • 'S' for Skip binary files
  • 'C' to Confirm each binary file

This flag is reset to the default after a /G. See Input filter.

/C<U|L|S|T|I|R...>
 Case translation, conversion, email and miscellaneous:
  • ‘U’ for UPPERCASE
  • ‘L’ for lowercase
  • ‘S’ for Sentence case
  • ‘T’ for Title Case
  • ‘I’ for tOGGLE cASE
  • 'R' for RanDOm caSE
  • 'M' for Convert IBM drawing characters
  • 'D' for debug filter
  • '-' for Delete All
  • 'Z' for Randomize lines
  • 'W' for Word list
  • 'V' for Reverse each line
  • '0' Extract email addresses
  • '1' UUencode
  • '2' UUdecode
  • '3' XXencode
  • '4' XXdecode
  • '5' Mime Base 64 encode
  • '6' Mime Base 64 decode
  • '7' Mime quoted printable encode
  • '8' Mime quoted printable decode
  • '9' Extract URLs

/CLEARCOMMENTS
 Clears all comments from the filter.
/CLEARFILES
 Clears all files from the file list.
/CLEARFILTERS
 Clears all filters from the filter list.
/D[A|B|D|E|H|K|L|R|S|W]...
 Removes:
  • 'L' Removes blank lines
  • 'E' Removes all spaces and tabs from the end of lines
  • 'S' Removes all spaces and tabs from the start of lines
  • 'B' Removes binary characters
  • ‘A’ Removes ANSI codes
  • 'D' Translate IBM drawing characters
  • 'H' Remove HTML and SGML
  • 'K' Remove backspaces
  • 'R' Resolve backspaces
  • 'W' Removes multiple whitespace
  • 'M' Removes email headers
/deletelistfile | Deletes the list file. Only works when placed immediately after a /L=listfile directive.
/EXPORTFILTERTOLOG | Exports the filter's text to the log file. Useful for command line automation to ensure that the filter you build via the command line is the filter you get. The log filename MUST have already been specified. Should only be used after the filter list has been completely built, otherwise you will only get partial results.
/EXTENSION=<new extension>
 Sets the new file extension to be given to all output files. See Output filter.
/E[R]<D|M|U|A|F<cpr>><D|M|U|N>
 Line feed conversion. See End of line conversion filter.

Optional R to remove extraneous line feeds.

Input format:

  • ‘M’ for Macintosh
  • ‘D’ for DOS
  • ‘U’ for Unix/Amiga
  • ‘A’ for Auto
  • ‘F’ for Fixed Length, followed by the number of characters per record (cpr).

Output format:

  • ‘M’ for Macintosh
  • ‘D’ for DOS
  • ‘U’ for Unix/Amiga
  • ‘N’ for None (ie remove line feeds)

e.g.
/ERDU to convert from Dos to Unix EOL, removing extra linefeeds
/EF742D to convert from fixed record of 742 characters to DOS

/F=<filter filename>
 Open filter list file.

This command opens a filter file. Opening a filter file DELETES any filters and overrides all options that have been specified by any previous command line parameters e.g.

textpipe.exe /ERDU "/f=c:\Program Files\TextPipe\My Filters\test.fll"

If you have already loaded a filter file but have not yet activated it with /G, any subsequent /F commands will APPEND its filters to the existing filter list. This can be used to combine multiple filters together, even those that have sub filters. Note that this is different behaviour to inserting using the GUI, which always occurs at the current cursor position.

/G
 Go! Begin a conversion thread with the specified options. You can specify extra options and follow them with /G again to start a second thread of execution. After starting a thread of execution, all options return to their default settings.

After a thread has been initiated with /G, the following actions occur (in ADDITION to all flags returning to their defaults):

  • The entire list of filters is cleared
  • The entire list of files is cleared

Note: The /G command must be on the right hand side of all the commands that make up a thread of filtering.

/H<A|O>
 Character set translation
  • A for ANSI to OEM
  • O for OEM to ANSI
  • N for ANSI to Unicode
  • C for Unicode to ANSI

/I
/I+
/I-
 Subfolder recursion. Default: On. Specifying /I toggles it, /I+ turns it on, /I- turns it off.

This option is NOT reset after a /G

/L=<file list filename>
 Load a list file containing a list of filenames and wildcards. The file should contain one file specification per line e.g.
C:\docs\*.txt;*.ans
D:\unix\*.*

...etc

Note: The filenames in the list file must NOT be surrounded by double quotes.

See also the /deletelistfile directive.

/LOG=<filename>
 Sends log information to the specified file. This is primarily used during command line automation. If the file cannot be written to or disk space fills up logging is turned off. TextPipe will force the creation of folders if necessary. See logging options.

The default file is 'textpipe.log' in the folder TextPipe was started from. A blank log filename turns logging off, any other value turns logging on.

/LOGENABLE=(0|1)
 Controls whether logging is on or off
  • 0 = Logging disabled
  • 1 - Logging enabled

See logging options.

/LOGMODE=(0|1)
 Controls the logging mode.
  • 0 = Overwrite existing file
  • 1 - Append to existing file

See logging options.

/LOGTHRESHOLD=<minimum lines>
 Controls the minimum number of log lines before the log data is written to the log file. A value of zero forces all lines to be written out. This is useful for preventing logging for small jobs, and keeps the size of the log file small.

See logging options.

/M=<map filename>
 Map - adds a filter with the specified (single byte) map file. See Maps. See also /UNICODEMAP
/MAXIMIZED
 TextPipe starts maximized.

This command line option MUST be placed FIRST.

/MERGETOFILE=<merge filename>
 Sets the output filter to merge all output to a single file, and sets the filename to the filename specified. See Output filter.
/MINIMIZED
 TextPipe starts minimized. TextPipe also uses less memory when minimized because the screen representation does not need to be stored. This command line option MUST be placed FIRST.
/MULTIMAP=<file.xls>
 Specifies the Excel document to load as a Unicode multi-map. By setting the MULTIMAPORDER, the same Excel spreadsheet can be used to map to different output formats.
/MULTIMAPLANGUAGECALL=<string>
 Specifies the format of the text string used to indicate a change of output language in a multi-map. The default is

((=%s=))

where %s is the new language name.

/MULTIMAPORDER=<Primary>(,secondary,etc)
 Specifies the precedence order given to the columns of an Unicode multi-map. The value is a comma-separated list of column names (each name is stored at the top of the column).

The first column specified is called the primary. If only the primary is specified, then the remaining columns are used in left-to-right order (ignoring the primary column). If two or more columns are specified, then only those columns are searched for values.

/N
/N+
/N-
 Retain the original file’s date and time. Default: Off. Specifying /N toggles it, /N+ turns it on, /N- turns if off. See Output filter.

This flag is reset to the default after a /G

/NEWFILTER=<new filter filename>
 Similar to /F, but opens a new filter and sets its filename before it gets saved.
/OUTPUTFOLDER=<new output folder>
 Sets the new output folder for all files to be written to. See Output filter.
/OUTPUTKEEPSTRUCT
/OUTPUTKEEPSTRUCT+
/OUTPUTKEEPSTRUCT-
 Controls whether the folder structure of the input files is retained when the output files get written to a new folder. See /OUTPUTFOLDER, and the Output filter.
/P<A|E,,,>
 Representation translation and other miscellaneous filters
  • A for ASCII to EBCDIC
  • E for EBCDIC to ASCII
  • D for decimal dump
  • H for hex dump
  • U for ROT13 encode/decode
  • 0 for hex encode
  • 1 for hex decode
  • 2 for HTTP encode
  • 3 for HTTP decode
  • 4 for reverse line order

/PLAYSOUND=(0|1)
 Controls whether a sound is played on completion
  • 0 = Sound is disabled
  • 1 - Sound is enabled (default)

/PREPAREFILTERFORDISTRIBUTION
 Prepares the filter for distribution. See the Tools Menu
/Q
 Exit after completion. Default: Off. Enabling this option causes TextPipe to exit immediately after all conversion threads complete. If an error occurs, it is displayed.

Note: The /Q command should be the very last command on the command line.

/R[C][P]=<replace string>
 Replace string. See Search and Replace filter.

Optional C for Case Sensitive replace. Default: Off. Select this option to use the same case for the replace string as was found in the search string.

Optional P to prompt on replace. Default Off.

Must be preceded by a /S to specify the search string. After the replace string has been specified, the Case Sensitive and Whole Words options both reset to Off.

/REPLACELIST=<filename>
 Loads a search/replace list filter with the specified file.
/S[C][W][X][0-6]=<search string>
 Search string. See Search and Replace filter.

You can also add a list of search and replace pairs through the user interface. See also Converting Search/Replaces from other sources.

Optional C for Case sensitive search/replace. Default: Off. Select this command to search for text that is capitalized exactly the way you type it in the Search text box. If you do not select Case, TextPipe finds all occurrences of the text, regardless of capitalization.

Optional W for whole words only (Default: Off)

Optional X to specify that the search string is a regular expression. Default off. This is equivalent to /S1, the old egrep-style pattern.

0-6 Search type. Default 0.

  • 0 - Exact search
  • 1 - Old egrep pattern
  • 2 - Sounds like
  • 3 - Edit distance
  • 4 - Perl pattern
  • 5 - Brief-style pattern
  • 6 - Word-style pattern
  • 7 - EasyPattern

/S... is used in conjunction with the /R command to specify the replace string. For strings with embedded spaces, surround the whole parameter in quotes e.g.

textpipe.exe "/SC=roses are blue" "/R=violets are yellow"

/SAVEFILTER=<filter filename>
 Saves the filter to the given filename. Any existing file is overwritten. The filename is added to the list of recent filters. TextPipe will force the creation of folders if necessary.
/ST<spaces per tab>
 Compresses spaces to tabs. See Spaces to Tabs filter.
/T
/T+
/T-
/TB
 Test mode. Default: Off. Specifying /T toggles it, /T+ turns it on, /T- turns if off, /TB enables backup mode. See Output filter.

This flag is reset to the default after a /G

/TS<spaces per tab>
 Expands tabs to spaces. Each tab is replaced by from 0 to Spaces Per Tab-1 spaces depending on the current column. See Tabs to Spaces filter.
/UNICODEMAP=<unicode map file1;file2;file3>
 Adds a filter with the specified (unicode) map file. Maps can be loaded from Excel worksheets (.XLS), Comma Separated Value files (.CSV, the default when the file extension is not recognized) and Tab delimited value files (.TAB). This command should be followed by the /UNICODEMAPOPTIONS command below, otherwise non-entered characters are passed through unchanged (Replace Type 0).

Multiple maps can be separated by semi-colons (;)

See Unicode maps. See also /M

/UNICODEMAPAPPEND=<unicode map filename>
 Appends a map file to the previously added Unicode Map. This command does NOT accept multiple map filenames.
/UNICODEMAPERRORS=<0|1>
 This command determines how overlapping regions are handled for the previously added Unicode map. When set to 0, overlapping regions are flagged as warnings (the default). When set to 1, overlapping regions are logged as errors.
/UNICODEMAPOPTIONS=<type>,<replace with>
 Sets the non-entered character options for the previously added Unicode Map filter.

Replace Type:

  • 0 - Pass through unchanged
  • 1 - Pass to sub filter
  • 2 - Replace with value (use next option to specify it)
  • 3 - Remove

Replace with:

The string to replace it with, if the Replace Type is 2.

e.g.

/UNICODEMAP=my_translations.xls
/UNICODEMAPOPTIONS=2,Invalid character found 0x%4.4x

See Unicode maps.

/UNICODEMAPUTFMODE=<0|1>
 Sets the UTF mode for the previously added Unicode Map filter.
  • 0 - UTF-16
  • 1 - UTF-32

See Unicode maps.

/WW<column width>
 Word wrap. See Word Wrap filter.
/Z=<filename>
 Read command line parameters from a command file. Very useful for creating filters based on Perl scripts or other search/replace programs, or for working around the Windows limitation of approx 260 characters on the command line. Note that you MUST remove the double quotes around command when you place them inside the command file. This option ignores blank lines in the command file, and allows comments on lines starting with a ';'.

For example, if you had the long command line

start /w textpipe.exe "/f=my special filter.fll" /g "/f=my special filter2.fll" /g "/f=my special filter3.fll" /g "/f=my special filter4.fll" /g /q

This would change to something like

start /w textpipe.exe "/z=commands.txt"

and the file commands.txt would contain

;my new command file
/f=my special filter.fll
/g
/f=my special filter2.fll
/g
/f=my special filter3.fll
/g
/f=my special filter4.fll
/g
;exit on completion
/q


Troubleshooting command-line automation

 Troubleshooting command-line automation

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This section describes how to troubleshoot command line automation.

  1. Check that /g and /q are at the VERY END of the command line. They affect all parameters specified to their left
  2. Verify that you have used EXACTLY the right format for the parameters - see the reference section
  3. Try removing the /g and /q and -
    1. Check that the correct filter file (.fll) is opened or that the correct list of filters appears
    2. Check that the correct files are displayed in the file list. Note that any incorrect, unrecognized or otherwise invalid command line parameters will be listed as filenames
    3. If no files are being processed - check that the Process Binary Files option is set correctly - the files may be being detected as binary files
    4. Check that the correct operation is performed when you click the Go button.


Exit codes

 Exit codes

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

TextPipe sets an exit code on completion that depends on the level of errors logged. Exit codes are useful for detecting program success from command line execution. You can use the DOS ERRORLEVEL command in a batch file to modify the execution flow according the exit code.

Highest logged error | Exit code
None | 0
Info | 1
Warning | 2
Error | 3

e.g. in a .BAT batch file.

textpipe.exe ....
IF ERRORLEVEL 3 do_something
IF ERRORLEVEL 2 do_something
IF ERRORLEVEL 1 do_something


System requirements

 System requirements

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

TextPipe requires Windows 95/98/2000/NT/ME/XP and a minimum of 16MB of main memory. The program requires approximately 5MB of memory when running. A full installation requires 3.5 MB disk space.


File Size Limitations

 File Size Limitations

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

TextPipe has NO upper limit on the size of files being processed - even files larger than 2GB can be processed or generated. Currently, only the Reverse line order filter requires the entire input file to be read into memory.

Several filters also require an entire ancillary file to be completed loaded into memory:

  • Retain lines found in
  • Remove lines found in

If you need to use these filters, you should try to keep the maximum file size to less than the total amount of physical RAM in your system. This ensures that system performance is still reasonable. For example, a system with 32 MB RAM should be Ok to Reverse Lines of files up to 30 MB in size.

Operating System Limitations

The maximum file size on a FAT32 formatted partition is 4 GB or 4,294,967,296 bytes minus 2 bytes.

The maximum file size on a FAT16 partition under Windows 9.x and ME is 2 GB minus 1 byte.

But the size limit using the FAT16 file system under Windows NT or 2K is 4 GB.

And the NTFS file system has a (theoretical) maximum file size of 16 exabytes or 18,446,744,073,709,551,616 bytes (2^64 sectors)


Defining Custom Patterns

 Defining Custom Patterns

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

All memo fields (multi-line text fields) and edit fields that support patterns also provide a menu item for Insert Custom Pattern. The patterns from this list are user-defined, and come from the file custom_patterns.txt in the same folder as TextPipe.exe.

The format for this file is as follows:

  • Comments can be placed on lines beginning with a semi colon
  • Blank lines are ignored
  • A pattern is introduced by a line beginning with '#'. The remainder of this line provides the pattern name which appears in the menu
  • All lines after the pattern name, up to a line starting with ###END###, are treated as part of a pattern.

Here is an example custom_patterns.txt file:

;User-Defined Patterns
;
;custom_patterns.txt - saved in TextPipe's folder (the same folder
;as where textpipe.exe resides)
;
;These patterns can be inserted from the right-click or context menu on
;any TextPipe field that supports patterns

#Internet URls
(FTP|HTTP)://([_a-z\d\-]+(\.[_a-z\d\-]+)+)(([_a-z\d\-\\\./]+[_a-z\d\-\\/])+)*
###END###

#Email addresses
\b([_a-z0-9-]+(\.[_a-z0-9-]+)*)@([_a-z0-9-]+(\.[_a-z0-9-]+)*)\.([a-z]{2,3})\b
###END###

#IP addresses
\b([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\b
###END###

#Credit card numbers
(\d{3,4})[- ]?(\d{4})[- ]?(\d{4})[- ]?(\d{4})
###END###


Optimizing Performance

 Optimizing Performance

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Disk I/O

The slowest operation that TextPipe performs is reading from and writing to disk. You can improve performance by making sure that all files being processed are stored on local disks rather than on network servers. You can also increase speed by an order of magnitude by using RAM drives - a disk held in memory, although naturally this won't help if the files you process are very large.

TextPipe utilizes specific Windows API calls to enhance the speed of reading data files.

Temporary files

TextPipe doesn't use temporary files at all, except for sorting where they are unavoidable. TextPipe only ever writes out the completed output file so far. It uses a file name like TXPxxx.tmp until the file is completely written out, then it renames it to the actual output filename.

If you have enough memory, the entire sort is performed in memory for speed. Every 10000 lines, TextPipe checks to see if there is less than 16MB of physical memory (not virtual memory!). If so, it writes the sorted results so far to a temporary file and then continues. If the Output Filter is set to File Output or Single File Output, any temporary files are written to the same folder as the output file. If the Output Filter is set to Clipboard Output then any temporary files are written to the current folder. All temporary files are removed as soon as possible during the sort progresses.

Pattern matching

You can improve matching performance by an order of magnitude by allowing patterns to fail earlier by limiting what wildcards like .* can match. If you can replace .* (match any character 0 or more times) with [^>]* (match any character except '>' 0 or more times) or [^>]{0,200} (match any character except '>' up to 200 times) then your patterns will match/fail to match far more quickly.

Greedy matching is almost always slower than non-greedy matching, and if the maximum allowed match size is very large then performance will slow to a crawl.

You should avoid patterns of the type

([^">])+

where the brackets are unnecessary. Each set of brackets causes a recursive step. This pattern should be written as:

[^">]+

When the match can occur a large number of times, it can often be rewritten as

[^">]{1,10} or ([^">]){1,10}

with no change to the pattern matching process. This avoids too many recursive steps.

Use Your Head!

See how many problems you can spot with this filter:

|--Restrict to each line in turn
| |
| |--Convert End of Lines - Auto to DOS
| | [X] Remove bad EOL
| |
| |--Replace [|] with []
| | [ ] Match case
| | [ ] Whole words only
| | [ ] Case sensitive replace
| | [ ] Prompt on replace
| | [ ] Skip prompt if identical
| | [ ] First only
| | [ ] Extract matches
| |
| |--Replace [)] with []
| | [ ] Match case
| | [ ] Whole words only
| | [ ] Case sensitive replace
| | [ ] Prompt on replace
| | [ ] Skip prompt if identical
| | [ ] First only
| | [ ] Extract matches
| |
| +--Replace [)] with []
| [ ] Match case
| [ ] Whole words only
| [ ] Case sensitive replace
| [ ] Prompt on replace
| [ ] Skip prompt if identical
| [ ] First only
| [ ] Extract matches
|
|--Restrict to each line in turn
| |
| +--Insert column 0 [@77777777]
|
|--Restrict to each line in turn
| |
| +--Insert column 0 [@88888]
|
|--Restrict to each line in turn
| |
| +--Insert column 0 [@99999]
|
|--Replace [@] with [|]
| [ ] Match case
| [ ] Whole words only
| [ ] Case sensitive replace
| [ ] Prompt on replace
| [ ] Skip prompt if identical
| [ ] First only
| [ ] Extract matches

<-- This restriction filter is unnecessary, all the filters inside it already effectively restricted to each line

1. Since the replacement text for Replacement #1, #2 and #3 is the same (blank), we could combine all three and use a
perl pattern searching for:  [()|]    OR
an EasyPattern searching for:  [ '(' or ')' or '|' ]

2.

3.

<-- This restriction filter is unnecessary, the insert column filter does not need it

1. Could be combined with #2 and #3 below, to insert all columns at the same time

<-- This restriction filter is unnecessary, the insert column filter does not need it

2. See #1

<-- This restriction filter is unnecessary, the insert column filter does not need it

3. See #1

<-- Could have been done in #1, #2 and #3 above, removing the need for a search/replace filter.

Scripting

Use of the scripting filter will slow TextPipe down because of the extra COM overhead of passing data to and from the script, and the script code being interpreted rather than compiled. Ensure that the code you write is as efficient as possible. You may be able to avoid the use of a script by judicious use of patterns, but this may be more complicated to understand. Use comments liberally - they do not slow processing down at all.

TextPipe Engine

If performance is key, you can take advantage of our embeddable TextPipe Engine. While the filter setup is done via scripting, the actual filtering is done with no GUI feedback, and no error logging for maximum performance.

Professional Services

Our consulting team offer filter optimization as a service. Please see our website for details.


Null characters - Clipboard Limitations

 Null characters - Clipboard Limitations

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Clipboard Limitations

The Windows clipboard cannot be used to process text that contains null characters (ASCII code 0). This is because the clipboard contents are defined as null terminated, and operations on it may halt prematurely. Note that this is a Windows limitation – not a TextPipe limitation. Binary data should be place in a file for TextPipe to process it.

Regular expression limitations

While TextPipe's Perl pattern matcher DOES allow nulls to be search for, TextPipe's egrep implementation does not allow null characters to be searched for.


Standard ASCII codes

 Standard ASCII codes

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

ASCII = American National Standard Code for Information Interchange

Quoted-Printable encoding uses 3 digit decimal code prefixed by \ eg. CR is \013.

Decimal | Code | ASCII
0 | NUL | Null
1 | SOH | Start of Heading (CC)
2 | STX | Start of Text (CC)
3 | ETX | End of Text (CC)
4 | EOT | End of Transmission (CC)
5 | ENQ | Enquiry (CC)
6 | ACK | Acknowledge (CC)
7 | BEL | Bell
8 | BS | Backspace (FE)
9 | HT | Horizontal Tabulation (FE)
10 | LF | Line Feed (FE)
11 | VT | Vertical Tabulation (FE)
12 | FF | Form Feed (FE) or Page Eject
13 | CR | Carriage Return (FE)
14 | SO | Shift Out
15 | SI | Shift In
16 | DLE | Data Link Escape (CC)
17 | DC1 | Device Control 1
18 | DC2 | Device Control 2
19 | DC3 | Device Control 3
20 | DC4 | Device Control 4
21 | NAK | Negative Acknowledge (CC)
22 | SYN | Synchronous Idle (CC)
23 | ETB | End of Transmission Block (CC)
24 | CAN | Cancel
25 | EM | End of Medium
26 | SUB or EOF | Substitute or End of File
27 | ESC | Escape
28 | FS | File Separator (IS)
29 | GS | Group Separator (IS)
30 | RS | Record Separator (IS)
31 | US | Unit Separator (IS)
32 | SP | Space
33 | ! | Exclamation Point
34 | " | Quotation Mark
35 | # | Number Sign, Octothorp, "pound"
36 | $ | Dollar Sign
37 | % | Percent
38 | & | Ampersand
39 | ' | Apostrophe, Prime
40 | ( | Left Parenthesis
41 | ) | Right Parenthesis
42 | * | Asterisk, "star"
43 | + | Plus Sign
44 | , | Comma
45 | - | Hyphen, Minus Sign
46 | . | Period, Decimal Point, "dot"
47 | / | Slash, Virgule
48 | 0 | 0
49 | 1 | 1
50 | 2 | 2
51 | 3 | 3
52 | 4 | 4
53 | 5 | 5
54 | 6 | 6
55 | 7 | 7
56 | 8 | 8
57 | 9 | 9
58 | : | Colon
59 | ; | Semicolon
60 | < | Less-than Sign
61 | = | Equal Sign
62 | > | Greater-than Sign
63 | ? | Question Mark
64 | @ | At Sign
65 | A | A
66 | B | B
67 | C | C
68 | D | D
69 | E | E
70 | F | F
71 | G | G
72 | H | H
73 | I | I
74 | J | J
75 | K | K
76 | L | L
77 | M | M
78 | N | N
79 | O | O
80 | P | P
81 | Q | Q
82 | R | R
83 | S | S
84 | T | T
85 | U | U
86 | V | V
87 | W | W
88 | X | X
89 | Y | Y
90 | Z | Z
91 | [ | Opening Bracket
92 | \ | Reverse Slant
93 | ] | Closing Bracket
94 | ^ | Circumflex, Caret
95 | _ | Underline, Underscore
96 | ` | Grave Accent
97 | a | a
98 | b | b
99 | c | c
100 | d | d
101 | e | e
102 | f | f
103 | g | g
104 | h | h
105 | i | i
106 | j | j
107 | k | k
108 | l | l
109 | m | m
110 | n | n
111 | o | o
112 | p | p
113 | q | q
114 | r | r
115 | s | s
116 | t | t
117 | u | u
118 | v | v
119 | w | w
120 | x | x
121 | y | y
122 | z | z
123 | { | Opening Brace
124 | | | Vertical Line
125 | } | Closing Brace
126 | ~ | Tilde
127 | DEL | Delete
128 |  | Reserved
129 |  | Reserved
130 |  | Reserved
131 |  | Reserved
132 | IND | Index (FE)
133 | NEL | Next Line (FE)
134 | SSA | Start of Selected Area
135 | ESA | End of Selected Area
136 | HTS | Horizontal Tabulation Set (FE)
137 | HTJ | Horizontal Tabulation with Justification (FE)
138 | VTS | Vertical Tabulation Set (FE)
139 | PLD | Partial Line Down (FE)
140 | PLU | Partial Line Up (FE)
141 | RI | Reverse Index (FE)
142 | SS2 | Single Shift Two (1)
143 | SS3 | Single Shift Three (1)
144 | DCS | Device Control String (2)
145 | PU1 | Private Use One
146 | PU2 | Private Use Two
147 | STS | Set Transmit State
148 | CCH | Cancel Character
149 | MW | Message Waiting
150 | SPA | Start of Protected Area
151 | EPA | End of Protected Area
152 |  | Reserved
153 |  | Reserved
154 |  | Reserved
155 | CSI | Control Sequence Introducer (1)
156 | ST | String Terminator (2)
157 | OSC | Operating System Command (2)
158 | PM | Privacy Message (2)
159 | APC | Application Program Command (2)
160-254 |  |
255 |  | Invisible space
NOTES:
(1) Introducer
(2) String Delimiter
(CC) Communication Control
(FE) Format effector
(IS) Information Separator

Unprintable Characters


Decimal to Hex Conversion Chart

 Decimal to Hex Conversion Chart

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

 | Second hex digit (least significant, or rightermost)
First hex digit | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | A | B | C | D | E | F
0 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15
1 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31
2 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47
3 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63
4 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79
5 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95
6 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111
7 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127
8 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143
9 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 155 | 156 | 157 | 158 | 159
A | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175
B | 176 | 177 | 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | 187 | 188 | 189 | 190 | 191
C | 192 | 193 | 194 | 195 | 196 | 197 | 198 | 199 | 200 | 201 | 202 | 203 | 204 | 205 | 206 | 207
D | 208 | 209 | 210 | 211 | 212 | 213 | 214 | 215 | 216 | 217 | 218 | 219 | 220 | 221 | 222 | 223
E | 224 | 225 | 226 | 227 | 228 | 229 | 230 | 231 | 232 | 233 | 234 | 235 | 236 | 237 | 238 | 239
F | 240 | 241 | 242 | 243 | 244 | 245 | 246 | 247 | 248 | 249 | 250 | 251 | 252 | 253 | 254 | 255


Unprintable Characters

 Unprintable Characters

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

TextPipe allows you to embed control characters in search and replace strings, line prefixes and user-defined replacement strings. All fields that support entering unprintable characters in this manner have a "context" or right-click menu that allows them to be entered without remembering the exact code.

E.g. To search for "hello" followed by a tab followed by "there" followed by a DOS end of line, enter the search string as

hello\tthere\r\n

where \t is the code for a Tab (see Standard ASCII codes).

Other Special Codes

Code | ASCII value | Description
\a | 7 | Bell
\b | 8 | Backspace
\cC |  | Control character. ie \cA inserts a Ctrl+A, \cM inserts a Ctrl+M
\t | 9 | Horizontal tab
\n | 10 | Line feed
\v | 11 | Vertical tab. You can also use \x0b, perl patterns do NOT support \v
\f | 12 | Form feed
\r | 13 | Carriage return
\e | 27 | Escape
\xXX | Any | Any character code, specified in hex. The XX is replaced by the character's 2 digit hex code
\x{ .. } | Any | Any sequence of character codes, specified in hex. This can be very handy for specifying unicode code sequences, such as \x00000020 for a unicode space character, instead of \x00\x00\x00\x20. To enter a two-byte number, use \x{0020}. For 3 bytes, use \x{000020}. For 4 bytes use \x{00000020}.


How can I use TextPipe?

 How can I use TextPipe?

Submit feedback on this topic

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home

Its myriad uses include:

  • Performing multiple search/replaces on a batch of files - very useful for Web sites.
  • Reducing the size of HTML files for faster page loading
  • Making HTML files readable without a Web browser
  • Converting or reading mainframe, Unix, DOS, Amiga or Macintosh files
  • Reformatting text e.g. preparing ASCII files for word processing
  • Convert mainframe reports to PC format and copy business user’s PCs overnight in a single step
  • Extract data from cumbersome log files
  • Organize lists
  • Customize text output from other programs
  • Reformat program source code
  • Automate manual processes
  • Interface incompatible software systems
  • Convert exported text for use by other software packages
  • Format mailing lists for printing
  • Search for and/or remove duplicates
  • Convert tabular data to delimited and vice versa
  • Cross-platform development
  • Adding extra word-processing capabilities through converting the clipboard contents
  • Making text files readable that contain ANSI escape codes, binary characters or IBM graphics characters
  • Adding an email reply prefix to every line
  • Converting text from one language to another (e.g. Pascal to c)
  • Correcting text files incorrectly FTP'd in binary mode.

See also

What is TextPipe?
 Features
 New features in this release


 Format Specifiers

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Format Specifiers

Format strings passed to the string formatting routines contain two types of objects--plain characters and format specifiers. Plain characters are copied verbatim to the resulting string. Format specifiers fetch arguments from the argument list and apply formatting to them. In the case of the Add Line Numbers filter, the only argument is the current line number.

Format specifiers have the following form:

"%" [index ":"] ["-"] [width] ["." prec] type

A format specifier begins with a % character. After the % come the following, in this order:

  • An optional argument index specifier, [index ":"]
  • An optional left justification indicator, ["-"]
  • An optional width specifier, [width]
  • An optional precision specifier, ["." prec]
  • The conversion type character, type

The following table summarizes the possible values for type:

d Decimal. The argument must be an integer value. The value is converted to a string of decimal digits. If the format string contains a precision specifier, it indicates that the resulting string must contain at least the specified number of digits; if the value has less digits, the resulting string is left-padded with zeros.

e Scientific. The argument must be a floating-point value. The value is converted to a string of the form "-d.ddd...E+ddd". The resulting string starts with a minus sign if the number is negative. One digit always precedes the decimal point.
The total number of digits in the resulting string (including the one before the decimal point) is given by the precision specifier in the format string--a default precision of 15 is assumed if no precision specifier is present. The "E" exponent character in the resulting string is always followed by a plus or minus sign and at least three digits.

f Fixed. The argument must be a floating-point value. The value is converted to a string of the form "-ddd.ddd...". The resulting string starts with a minus sign if the number is negative.
The number of digits after the decimal point is given by the precision specifier in the format string--a default of 2 decimal digits is assumed if no precision specifier is present.

g General. The argument must be a floating-point value. The value is converted to the shortest possible decimal string using fixed or scientific format. The number of significant digits in the resulting string is given by the precision specifier in the format string--a default precision of 15 is assumed if no precision specifier is present.
Trailing zeros are removed from the resulting string, and a decimal point appears only if necessary. The resulting string uses fixed point format if the number of digits to the left of the decimal point in the value is less than or equal to the specified precision, and if the value is greater than or equal to 0.00001. Otherwise the resulting string uses scientific format.

n Number. The argument must be a floating-point value. The value is converted to a string of the form "-d,ddd,ddd.ddd...". The "n" format corresponds to the "f" format, except that the resulting string contains thousand separators.

m Money. The argument must be a floating-point value. The value is converted to a string that represents a currency amount. The conversion is controlled by the CurrencyString, CurrencyFormat, NegCurrFormat, ThousandSeparator, DecimalSeparator, and CurrencyDecimals global variables, all of which are initialized from the Currency Format in the International section of the Windows Control Panel. If the format string contains a precision specifier, it overrides the value given by the CurrencyDecimals global variable.

p Pointer. The argument must be a pointer value. The value is converted to a string of the form "XXXX:YYYY" where XXXX and YYYY are the segment and offset parts of the pointer expressed as four hexadecimal digits.

s String. The argument must be a character, a string, or a PChar value. The string or character is inserted in place of the format specifier. The precision specifier, if present in the format string, specifies the maximum length of the resulting string. If the argument is a string that is longer than this maximum, the string is truncated.

x Hexadecimal. The argument must be an integer value. The value is converted to a string of hexadecimal digits. If the format string contains a precision specifier, it indicates that the resulting string must contain at least the specified number of digits; if the value has fewer digits, the resulting string is left-padded with zeros.

Conversion characters may be specified in uppercase as well as in lowercase--both produce the same results.
For all floating-point formats, the actual characters used as decimal and thousand separators are obtained from the DecimalSeparator and ThousandSeparator global variables.

Index, width, and precision specifiers can be specified directly using decimal digit string (for example "%10d"), or indirectly using an asterisk character (for example "%*.*f"). When using an asterisk, the next argument in the argument list (which must be an integer value) becomes the value that is actually used. For example,

Format('%*.*f', [8, 2, 123.456])

is the same as

Format('%8.2f', [123.456]).

A width specifier sets the minimum field width for a conversion. If the resulting string is shorter than the minimum field width, it is padded with blanks to increase the field width. The default is to right-justify the result by adding blanks in front of the value, but if the format specifier contains a left-justification indicator (a "-" character preceding the width specifier), the result is left-justified by adding blanks after the value.

An index specifier sets the current argument list index to the specified value. The index of the first argument in the argument list is 0. Using index specifiers, it is possible to format the same argument multiple times. For example "Format('%d %d %0:d %d', [10, 20])" produces the string '10 20 10 20'.


Macros

 Macros

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Macros allow you to insert information that varies while the filter runs - e.g. the name of the current file being processed. Macro names are not case sensitive.

Macro Name | Description
@inputFilename Replaced by the name of the current file being processed. The drive and path section of the filename is removed. Usually used with the Add text at start of file filter to place the name of the file being processed at the beginning, especially if output is being merged to one file or to the clipboard e.g.
\013\010-- This is file @inputFilename --\013\010
@fullInputFilename | Same as @inputFilename, except it includes the complete path
@clipboard | Inserts the contents of the Windows clipboard
@time | Inserts the time with the system-defined long time format
@date | Inserts the date with the system-defined short date format
@datetime | Inserts the date and time with the system-defined short date format and long time format.

All fields that support macros have a "context" or right-click menu that allows them to be entered without remembering the exact code.

If you need to use the actual name of a macro in your text e.g. '@inputFilename', simply double the initial @ i.e. '@@inputFilename' to force TextPipe to ignore it.

The following filters allow the use of macros to insert predefined text.

Add text at start of file
Add text at start of lines
Add text at end of lines
Add text to end of file
Add Column of Text


Unattended install

Unattended Installation

The Setup program accepts optional command line parameters. These can be useful to system administrators, and to other programs calling the Setup program.

/SP- Disables the This will install... Do you wish to continue? prompt at the beginning of Setup. Of course, this will have no effect if the DisableStartupPrompt [Setup] section directive was set to yes.

/SILENT, /VERYSILENT Instructs Setup to be silent or very silent. When Setup is silent the wizard and the background window are not displayed but the installation progress window is. When a setup is very silent this installation progress window is not displayed. Everything else is normal so for example error messages during installation are displayed and the startup prompt is (if you haven't disabled it with DisableStartupPrompt or the '/SP-' command line option explained above)

If a restart is necessary and the '/NORESTART' command isn't used (see below) and Setup is silent, it will display a Reboot now? messagebox. If it's very silent it will reboot without asking.

/NORESTART Instructs Setup not to reboot even if it's necessary.

/LOADINF="filename" Instructs Setup to load the settings from the specified file after having checked the command line. This file can be prepared using the '/SAVEINF=' command as explained below.

Don't forget to use quotes if the filename contains spaces.

/SAVEINF="filename" Instructs Setup to save installation settings to the specified file. Don't forget to use quotes if the filename contains spaces.

/DIR="x:\dirname" Overrides the default directory name displayed on the Select Destination Directory wizard page. A fully qualified pathname must be specified. If the [Setup] section directive DisableDirPage was set to yes, this command line parameter is ignored.

/GROUP="folder name" Overrides the default folder name displayed on the Select Start Menu Folder wizard page. If the [Setup] section directive DisableProgramGroupPage was set to yes, this command line parameter is ignored.

/NOICONS Instructs Setup to initially disable the Don't create any icons check box on the Select Start Menu Folder wizard page.

/COMPONENTS="comma separated list of component names" Overrides the default components settings. Using this command line parameter causes Setup to automatically select a custom type.


Make Big Endian

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced

About libiconv

libiconv is the GNU library used by TextPipe for its Unicode conversions.

C-Source code, .obj files and binaries are available for free from

http://www.gnu.org/software/libiconv/


Unicode Filters

 Unicode Filters

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

These filters perform a variety of Unicode functions.

Unicode conversion (Pro only)
ANSI to Unicode (Pro only)
Unicode to ANSI (Pro only)
Remove Unicode BOM (Byte Order Mark) (Pro only)
Swap UTF-16 word order (Pro only)
Swap UTF-32 word order (Pro only)
Make Big Endian (Pro only)
Make Little Endian (Pro only)

These filters were previously found in the Convert Menu.


Preferences Dialog

 Preferences Dialog

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Play Sound On Completion

When enabled, the specified sound is played when a TextPipe job completes. This is useful for long jobs - to let you know when they finish.

Sound to Play

The name of the sound file (.wav) to play. The browse button can be used to find the file.

Default Editor (if no association found)

When a filename is double clicked within the file list or in the results pane of the status dialog, the editor specified here will be used if no association can be found for the file. The default is Windows Notepad.

Show Tips of the Day

When enabled, Tips of the Day are shown when TextPipe begins.

File History Entries

The number of recently used filters to display in the file menu.

Language

This item allows you to change the display language used by TextPipe. To translate TextPipe into your language, or to find what translations are available, please visit http://www.crystalsoftware.com.au/languages.html

Font for Trial Run Areas

The font to be used for displaying text in the trial input and trial output areas. To change the font, click the Change button.

Ok

Saves changes and closes the window.

Cancel

Discards changes and closes the window.


COM examples

 COM examples

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

The following demonstrations show how TextPipe can be controlled from a web page using VBScript or JScript. You could easily customise this page with your own logo, instructions, links to your website and much more, and then deliver it to your clients as a custom front-end. The script can also be encrypted so that users can't change it. We can also provide installation support so that you can package the front-end and TextPipe Pro for your clients.

Demo 1 - Add header/footer/left & right margins

Input data

Add this header
Add this left margin
 Add this right margin
Add this footer

(Output data is shown below)

Output data

Demo 2 - Add filters, leave TP running

This demonstration starts TextPipe, adds a dummy filter, and then leaves TextPipe open for running.

Source code

Below is the source code embedded in this page:

<script language="VBScript">

' addStringFilter type - the type of filter to add
stftAddLeftMargin = 0
stftAddHeader = 1
stftAddFooter = 2
stftAddRightMargin = 3
sftConvertToRandomCase = 42

dim TextPipeApp2

function test()

dim Filter1
dim TextPipeApp
Set TextPipeApp = CreateObject("TextPipe.Application")

Set Filter1 = TextPipeApp.newWindow
Filter1.startFilters
Filter1.clearAllFilters
Filter1.addCommentFilter( "This filter list was created from VBScript on
a webpage" )

if form1.leftmargin.value <> "" then a = Filter1.addStringFilter(
stftAddLeftMargin, form1.leftmargin.value )
if form1.rightmargin.value <> "" then a = Filter1.addStringFilter(
stftAddRightMargin, form1.rightmargin.value )
if form1.header.value <> "" then a = Filter1.addStringFilter(
stftAddHeader, form1.header.value )
if form1.footer.value <> "" then a = Filter1.addStringFilter(
stftAddFooter, form1.footer.value )
Filter1.endFilters

form1.output.value = Filter1.processString( form1.input.value )

'close TextPipe
'if you don't want to close TextPipe, omit the closeWithoutSave line
filter1.closeWithoutSave
Set Filter1 = Nothing
Set TextPipeApp = Nothing

end function

function test2()

dim Filter1
Set TextPipeApp2 = CreateObject("TextPipe.Application")

Set Filter1 = TextPipeApp2.newWindow
Filter1.startFilters
Filter1.clearAllFilters
Filter1.addCommentFilter( "This filter list was created from VBScript on
a webpage" )

Filter1.addSimpleFilter( sftConvertToRandomCase ) 'randomize case
Filter1.endFilters

end function

function test3()

Set TextPipeApp2 = Nothing

end function

</script>


Converting Search/Replaces from other sources

 Converting Search/Replaces from other sources

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

If you have search/replace pairs in Perl script or text files from other Search/Replace packages, you can use a simple procedure to convert them to TextPipe. You can also add a list of search and replace pairs through the user interface.

Edit the file into the following form

/S=<search string>
/R=<replace string>
/S=<search string 2>
/R=<replace string 2>
/S=<search string 3>
/R=<replace string 3>

..etc

You could even use TextPipe to do this for you!

Save this edited script to a file (call it import.txt). Then start TextPipe as follows:

textpipe.exe /z=import.txt

This effectively reads command-line parameters from a file instead of from the command line. Quotes are not required for search/replace strings with embedded spaces.

If you know that the original source uses Regular Expressions, then you can replace the

/S=<search string>

with

/SX=<pattern string>

Typically, regular expression matches are case-sensitive, so you could also use

/SCX=<case-sensitive pattern>

to ensure the original search/replace is unchanged.

This is the exact procedure used to import TextPipe's Jive, Kraut and Elmer Fudd filters from a variety of sources.


Popup menus

 Popup menus

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up
Defining Custom Patterns

Popup menus (context menus)

All memo fields (multi-line text fields) and most edit fields support popup menus. Right click in the field to active the menu. TextPipe lists the applicable commands, and dims any commands that are not possible (e.g. if the clipboard is empty, is makes no sense to allow Paste).

Insert special character

TextPipe uses special notation to define characters that cannot be entered from the keyboard, such as a form feed, represented by '\f'. This menu gives you a list of options to insert a variety of special characters in the field. Depending on the type of field (e.g. Exact match, Pattern match), different characters will have a special meaning, and so the menu that is shown will change.

Insert control character

These menu items are used to insert control characters directly, without using special codes to represent them e.g. when a tab is inserted, an ASCII code 9 is inserted, not \t.

Insert pattern

Fields that support pattern matching will also display a submenu to help you quickly enter patterns for common situations.

Insert custom pattern

Fields that support pattern matching will also display a submenu allowing user-defined patterns to be entered. For more details on setting this up, please see defining custom patterns.

Insert macro

For edit fields that support macros (@inputFilename etc), this menu will automatically add one of the supported macro names.

Escape special characters

As mentioned above, TextPipe uses special notation to define characters that cannot be entered from the keyboard. Use this command to ensure that TextPipe cannot misinterpret any control characters that you may inadvertently have in the edit field e.g. backslash characters are doubled \ -> \\

Allow extra spaces

This command converts the selected text of a search pattern (or all text if there is no selection) into one capable of absorbing extra spaces or tabs found in the search string. This can be very handy for matching in the HTML documents of a website where the HTML editor may have inserted extra spaces or tabs.

This command is only available when the selected field is the Search field of a search pattern.

Allow extra line feeds and spaces

This command converts the selected text of a search pattern (or all text if there is no selection) into one capable of absorbing extra spaces, tabs or line feeds found in the search string. This can be very handy for matching in the HTML documents of a website where the HTML editor may have inserted extra spaces, tabs or line feeds.

This command is only available when the selected field is the Search field of a search pattern.

Allow RTF tags

This command converts the selected text (or all text if there is no selection) into one capable of absorbing RTF tags. RTF tags are present in a .RTF (Rich Text Format) files. This facility is almost essential for searching and replacing inside RTF files.

The Allow RTF tags feature is only designed for use with text that consists of normal words - not with text that already contains a pattern.

This command is only available when the selected field is the Search field of a search pattern.

Find

The Find command searches for text in the field.

Find next

The Find Next command searches for the next occurrence of the previous text searched for. TextPipe will beep if there is no next occurrence, and if Find Next is activated again it will recommence the search at the beginning of the field.

Word wrap

Toggles word wrapping for the memo control. This property is not saved - it just makes it easier to view text.

Show hex dump

Shows a hex dump of the selected text or the entire text if there is no selection. This is handy to find out exactly what control characters are present in a control.


Prompt On Replace

 Prompt On Replace

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

The prompt on replace dialog is shown when a search filter with "Confirm before replace" checked finds a match.

promptReplace.gif (5790 bytes)

The Found Text tab displays the matching text with a highlight, and where possible, also shows surrounding text so that you see the context of the match. If the replacement text is not appropriate, you can edit it. Where the search filter is NOT a regular expression, changes to the replacement text are used as default replacement text for all future matches. A right-click or context menu lets you insert special characters in the replacement text field.

The Search Criteria tab shows the search string or pattern, and if the search filter IS a regular expression match, it also shows the contents of the $0-$9,$a-$z variables for the match (depending on the number of bracketed expressions in the pattern match expression).

Yes | Replace this match. Continue prompting for remaining matches
No | Do not replace this match. Continue prompting for remaining matches
Yes (file) | Replace this match, and all others in this file. Resume prompting at the beginning of the next file.
No (file) | Don't replace this match, or any others in this file. Resume prompting at the beginning of the next file.
Yes (job) | Replace this match, and any others in every file.
No (job) | Don't replace this match, or any others in any file.
Cancel | Stops processing
Yes (subset) | When the text has been extracted by a sub filter, two extra buttons are displayed for replacing all in the remainder of the subset, and skipping all in the remainder of the subset.
No (subset) | When the text has been extracted by a sub filter, two extra buttons are displayed for replacing all in the remainder of the subset, and skipping all in the remainder of the subset.


Registry entries

 Registry entries

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This page documents the TextPipe registry settings that are required to interface to it.

HKCU\SOFTWARE\Crystal Software\TextPipe

showTips

0=disable tips, 1=show tips. Set this value to 0 to prevent the Tip of the Day window displaying.

playSound

0=no sound on filter completion, 1=play sound on filter completion.

soundFile

The sound file (.wav) to play if playSound=1.

isFirstDay

0=this is not the first time TextPipe is run,1=if this is the first TextPipe is run. Set this value to 0 to prevent the help window displaying when TextPipe is run.

updateDate

Gives the next date that TextPipe should check for an update. To disable the update check, set it way in the future with 41000 (i.e. after 2010).

HKLM\SOFTWARE\Crystal Software\TextPipe

rootDir

This key tells you where TextPipe is installed. TextPipe has not been installed if this key does not exist. The value of this key is TextPipe's installation folder. It is NOT guaranteed to end in a backslash, so you may have to add one manually.

TextPipeType

TextPipe comes in a number of different versions. Use this value to detect which version of TextPipe is installed and only display appropriate options. Currently these settings are only used by askSam.

TextPipeType | Internal reference | Functionality
1 | appTextPipePro | Interface enabled with editing
2 | appTextPipeStandard | Interface disabled
14 | appTextPipeProRunTime | Interface enabled without editing
18 | appTextPipeWeb | Interface disabled
20 | appTextPipeAskSam | Interface enabled with editing
21 | appTextPipeRunTimeAskSam | Interface enabled without editing

Name

Organization

Receipt

The name, organization and receipt number of this customer.


Regular Expression Hints

 Regular Expression Hints

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up
Perl Regular Expressions

Introduction

Need we say that regular expressions are a powerful, but sometimes tricky, part of TextPipe. The good news is that you don't need to be a rocket scientist to use regular expressions. The simple ones are just that -- fairly simple -- and they are all you need to understand for many applications. If you're relatively unfamiliar with regular expressions -- regexps for short -- be sure to read the section below Tips on Seat-of-the-Pants Pattern Matching.

If, on the other hand, you are already comfortable using regular expressions, you won't find much of value in the Tips section, but may find subsequent tables and examples useful. Our focus is on items, like quantifiers, that you keep looking up in your reference materials because you don't use them often enough to remember the fine points.

Tips on Seat-of-the-Pants Pattern Matching

These are tips, not a tutorial. We assume you already know how to code simple substitutions with regular expressions.

The irony of pattern matching with regular expressions is that although it's a highly deterministic operation, this doesn't help you much unless you are a rocket scientist. The pattern matching engine uses a complex set of rules. But, the rules are hard to understand and difficult to apply to practical problems of some complexity -- even for full-time Web page developers, much less part-timers. Fortunately, you can mostly forget about the formal matching rules and take a much more practical or seat-of-the pants approach.

With this as background, here are our tips:

  1. Never overwrite your input file. There's no Undo function as there is with your favorite word processor. It's easy, oh so easy, to make coding errors in substitution statements, even when you have experience coding them.
  2. If you're using the don't-care character sequence (".*" or ".+") in your pattern, watch out for greedy matching -- which typically matches the right-most pattern rather than the left-most. See the section Quantifiers for examples.

Metacharacters

The following characters -- so-called metacharacters -- have special meaning in regular expressions. To match them, precede them with a backslash.

Metacharacter | Purpose
\ | first character of assertions, such as \w or \d
| | OR; allows matching options
. | match any character
( ... ) | grouping operator; builds $1, $2, etc.
[ ... ] | match any character within brackets
+ | quantifier: match one or more times
? | quantifier: at most one match
* | quantifier: match zero or more times

Assertions identify special conditions for a pattern match. The following table covers commonly used assertions.

Common Assertions
Assertion | Function
\d | digit [0-9]
\d+ | number
\D | nondigit
\s | white-space
\S | non white-space
\w | alphanumeric character [a-zA-Z_0-9]
\w+ | alphanumeric word
\W | nonalphanumeric character

Quantifiers

This section complements discussions of quantifiers that are long on words but short on examples. We examine a host of variations on a basic example, using the various quantifiers in the following scenario:

Search for | (pe)
Replace with | --
Target string | Pepe pens Perl pearls.

We use parentheses in all examples for consistency. They only matter here when followed by a quantifier. Thus (pe)+ looks for successive occurrences of pe, whereas pe+ looks for successive occurrences of e.

Substitutions shown in red merit close attention.

Search string | Case sensitive? | Result | Notes
No quantifier (i.e., exactly one match) |  | Pepe pens Perl pearls.
(pe) | Y | Pe-- --ns Perl --arls.
(pe) | N | ---- --ns --rl --arls.
Plus quantifier (one or more matches) |  | Pepe pens Perl pearls.
(pe)+ | Y | Pe-- --ns Perl --arls.
(pe)+ | N | -- --ns --rl --arls.
Question mark quantifier (at most one match) |  | Pepe pens Perl pearls.
(pe)? | Y | --P--e---- ----n--s-- --P- etc.
Asterisk quantifier (zero or more matches) |  | Pepe pens Perl pearls.
(pe)* | Y | --P--e---- ----n--s-- --P- etc.
Leading don't-care characters |  | Pepe pens Perl pearls.
.*(pe) | Y | --arls. | Greedy match
.*(Pe) | Y | --rl pearls. | Greedy match
.+(Pe) | Y | --rl pearls.

See also

Regular expressions


Microsoft script control

 Microsoft script control

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

The Microsoft Script Control gives TextPipe the ability to process scripts written in VBScript and JScript, and with extra downloadable modules, PerlScript, REXXscript and PythonScript.

If the script control is not present or not working on your machine, please see the latest information at the Crystal Software site:

http://www.crystalsoftware.com.au/otherdownloads.html


TextPipe Sample Filter

 TextPipe Sample Filter

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Filter: $filter

$comments

This page demonstrates TextPipe being controlled from a web page using VBScript or JScript. You can easily customise this page with your own logo, instructions, links to your website and much more, and then deliver it to your clients as a custom front-end. The script can also be encrypted so that users can't change it. We can also provide installation support so that you can package the front-end and TextPipe Pro for your clients.

Uses VBScript to simply load and run the saved filter.
Uses VBScript to generate the filter from scratch, then runs it.
Uses JScript to simply load and run the saved filter.
Uses JScript to generate the filter from scratch, then runs it.

Input data (from Trial Run Input)

Output data

Contact Us Support Community Tutorials and User Guides (online)
© 1999-2005 Crystal Software . All rights reserved.


Multi-maps

 Multi-maps

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

A multi-map is a specialized version of a Unicode Map. It is used in font remapping applications to take a single input file, and map it to one or more output font sets. The hierarchy of the mapping can be controlled at run-time from the command line.

As each UTF-16 or UTF-32 character is processed, TextPipe first checks if it is marked as INVARIANT. If so, TextPipe scans languages in MULTIMAPORDER sequence for the first language with a non-blank entry, and outputs it without a language change sequence. If the character is not INVARIANT, TextPipe scans languages in MULTIMAPORDER sequence for the first language with a non-blank entry, outputting a language change sequence if required, the found value, and then resets the current language.

The Excel spreadsheet has individual UTF-16 Unicode hex code points in the left column (e.g. 0021, 003c), and a series of values in named columns or language families to the right (e.g. 01, 08, 09), starting at column 3. By setting the order that the languages are used (using MULTIMAPORDER), the same Excel spreadsheet can be used to map to different output languages.

Here is an example Excel multi-map:

UTF-16 | Invariant | 01 | 08 | 09 | 10 | 11 | 12 | 32 | 66
0009 | ON | \x09 | \x09 | \x09 | \x09 | \x09 | \x09 | \x09 | \x09
000A | ON | \x0A | \x0A | \x0A | \x0A | \x0A | \x0A | \x0A | \x0A
000D | ON | \x0D | \x0D | \x0D | \x0D | \x0D | \x0D | \x0D | \x0D
0020 | ON | \x20 | \x20 | \x20 | \x20 | \x20 | \x20 | \x20 | \x20
0021 |  | \x21 | \x21 | \x21 | \x21 | \x21 | \x21 | \x21 | \x21
003C |  | \x3C | \x3C | \x3C | \x3C |  |  | \x3C | \x3C
003D |  | \x3D | \x3D | \x3D | \x3D | \x3D |  | \x3D | \x3D
003E |  | \x3E | \x3E | \x3E | \x3E |  |  | \x3E | \x3E
003F |  | \x3F | \x3F | \x3F | \x3F | \x3F | \x3F | \x3F | \x3F
0040 |  | \x40 | \x40 | \x40 | \x40 | \x40 |  | \x40 | \x40
0041 |  | \x41 | \x41 | \x41 | \x41 | \x41 | \x41 |  |

The multi-map order command line options specifies the precedence order given to the columns. The first column specified is called the primary. If only the primary is specified, then the remaining columns are used in left-to-right order (ignoring the primary column). If two or more columns are specified, then only those columns are searched for values.

See also

Refresh map list
New map
Base map on filter list
Available maps


Remove byte range

 Remove byte range

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter is used to remove a range of bytes.

Several different means of specifying the bytes to remove are provided:

  1. Start byte, end byte. This removes all text including and between the specified byte.
  2. Start byte, width. Removes Width byte starting from (and including) the start byte.
  3. End byte, width. Removes Width fields backwards starting from (and including) byte End.
  4. Start byte to end of file. Removes all fields from the Start byte to the very end of the file.
  5. Width to end of file. Removes Width fields backwards starting from (and including) the last byte.

Note - if you are removing more than one byte range, it is easiest to remove ranges from right-to-left so that the position of the bytes doesn't change.

See also

Remove menu
Remove columns
Remove CSV fields
Remove Tab fields


Remove XML field

 Remove XML field

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter is used to remove HTML/XML-delimited elements, attributes, and text between tag pairs. A Remove All sub filter is automatically added.

The filter fields are identical to a HTML/XML restriction filter.

For example, if your data looked like this, and you wanted to remove the CompanyName field:

<TextPipeRow>
<CustomerID>ALFKI</CustomerID>
<CompanyName>Alfreds Futterkiste</CompanyName>
<ContactName>Maria Anders</ContactName>
<ContactTitle>Sales Representative</ContactTitle>
<Address>Obere Str. 57</Address>
<City>Berlin</City>
<Region></Region>
<PostalCode>12209</PostalCode>
<Country>Germany</Country>
<Phone>030-0074321</Phone>
<Fax>030-0076545</Fax>
</TextPipeRow>

Specify the 'Restrict to tag' name to remove as 'CompanyName'

See also

Remove menu
Remove columns
Remove delimited fields (CSV, Tab, Pipe etc)


Brief Pattern Search and Replace

 Brief Pattern Search and Replace

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter is identical in function to a pattern search and replace, except that it supports the Brief-editor syntax for specifying patterns:

Brief | TextPipe/perl pattern | Function
? | . | Match any character
@ | .* | Match zero or more characters
+ | .+ | Match one or more characters
< | ^ | Match the start of line
% | ^ | Match the start of line
$ | $ | Match the end of line
> | $ | Match the end of line
{ | ( | Start a group
} | ) | End a group


EasyPattern Search and replace

 EasyPattern Search and replace

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Use this filter to find a text pattern. You can also use this filter to find multi-line text. This filter uses the EasyPattern pattern matching language, which is very easy to use.

The length of each string is shown above it, and a warning is displayed on the status line if they are not the same. This can be very useful when replacing strings in binary files.

Find EasyPattern

The pattern to find. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy.

See also

  • EasyPattern pattern matching
  • EasyPattern reference

Pattern options button [...]

Maximum match size

Changes the maximum search buffer size. If you need to match a string that is longer than 4096 bytes (4k), increase the Maximum text buffer size to be at least as large as the string you need to match.

Match Case

When checked, the case of the string found must match the case of the Find Pattern exactly.

Find Whole Words Only

Only match the Find Pattern when it is not part of a larger word e.g. find "the" but not "them", "there", "worth" etc.

Replace With

The text to replace the found text with. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering unprintable and special characters and inserting pattern replacement specifiers:

$0 - is replaced by the entire matched expression
$1 - is replaced with the first parenthesised expression.
$2 - is replaced with the second parenthesised expression
$3 - $9, $a - $z and so on.

If you need to put a dollar sign $ in the Replace With section, simply double it i.e. $$.

The replacement text is passed to any sub filters.

Case Sensitive Replace

If the Find Pattern matches text that is entirely in UPPERCASE, this forces the replace string to all UPPERCASE. If the Find Pattern matches text that is entirely in lowercase, this forces the replace string to all lowercase.

Replace First Only

Only replace the first occurrence found - skip all remaining matches.

Prompt On Replace

By checking this option, you can manually verify each match before it gets replaced. You can choose to replace or ignore single matches, the remainder of the file or the entire job.

Skip Prompt If Identical

If the find string and replace string are identical, this option can be enabled to skip prompting. This can happen when the case of the search string is identical to the case of the replace string during a case-insensitive search.

Extract matches

When this option is checked all non-matching text will be discarded, leaving only the replacement text. This can be handy for data mining content from web sites or data files.


Edit Distance Search and Replace

 Edit Distance Search and Replace

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Use this filter to find a word that is spelt like another word, usually to locate a spelling mistake or a spelling variation.

An edit distance counts the minimum number of 'edits' required to change one word into another. A count of 1 is given to each character delete, character insert or swap of two adjacent characters. Extra deletes, inserts or swaps increase the edit distance between two words e.g.

Word 1 | Word 2 | Edit distance
Delphi | Delhi | 1 (delete)
Delhpi | Delphi | 1 (swap)
Dlphi | Delphi | 1 (insert)
Dlhpi | Delphi | 2 (insert + swap)

The length of each string is shown above it, and a warning is displayed on the status line if they are not the same. This can be very useful when replacing strings in binary files.

Find a word similar to

The single word to find. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy. No special characters are allowed.

Pattern options button [...]

Changes the edit distance threshold. Any strings with an edit distance less than this value will be replaced by the replace string.

Match Case

When checked, the case of the string found must match the case of the Find String exactly.

Find Whole Words Only

Disabled - always on by default.

Replace With

The text to replace the found text with. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, and entering unprintable and special characters.

To insert the found text into the replacement, use the text $0. If you need to put a dollar sign $ in the Replace With section, simply double it i.e. $$.

The replacement text is passed to any sub filters.

Case Sensitive Replace

If the Find String matches text that is entirely in UPPERCASE, this forces the replace string to all UPPERCASE. If the Find String matches text that is entirely in lowercase, this forces the replace string to all lowercase.

Replace First Only

Only replace the first occurrence found - skip all remaining matches.

Prompt On Replace

By checking this option, you can manually verify each match before it gets replaced. You can choose to replace or ignore single matches, the remainder of the file or the entire job.

Skip Prompt If Identical

If the find string and replace string are identical, this option can be enabled to skip prompting. This can happen when the case of the search string is identical to the case of the replace string during a case-insensitive search.

Extract matches

When this option is checked all non-matching text will be discarded, leaving only the replacement text. This can be handy for data mining content from web sites or data files.


Exact Search and replace

 Exact Search and replace

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Use this filter to find text that appears exactly as you have typed it, or with a case variation. You can also use this filter to find multi-line text.

The length of each string is shown above it, and a warning is displayed on the status line if they are not the same. This can be very useful when replacing strings in binary files.

Find String

The text to find. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, and entering unprintable and special characters.

Match Case

When checked, the case of the string found must match the case of the Find String exactly.

Find Whole Words Only

Only match the Find String when it is not part of a larger word eg find "the" but not "them", "there", "worth" etc.

Replace With

The text to replace the found text with. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, and entering unprintable and special characters.

To insert the found text into the replacement, use the text $0. If you need to put a dollar sign $ in the Replace With section, simply double it i.e. $$.

The replacement text is passed to any sub filters.

Case Sensitive Replace

If the Find String matches text that is entirely in UPPERCASE, this forces the replace string to all UPPERCASE. If the Find String matches text that is entirely in lowercase, this forces the replace string to all lowercase.

Replace First Only

Only replace the first occurrence found - skip all remaining matches.

Prompt On Replace

By checking this option, you can manually verify each match before it gets replaced. You can choose to replace or ignore single matches, the remainder of the file or the entire job.

Skip Prompt If Identical

If the find string and replace string are identical, this option can be enabled to skip prompting. This can happen when the case of the search string is identical to the case of the replace string during a case-insensitive search.

Extract matches

When this option is checked all non-matching text will be discarded, leaving only the replacement text. This can be handy for data mining content from web sites or data files.


Old Pattern (egrep) Search and Replace

 Old Pattern (egrep) Search and Replace

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Use this filter to find a text pattern. You can also use this filter to find multi-line text. This filter is slower and less flexible than using a perl pattern, but is maintained for backward compatibility.

The length of each string is shown above it, and a warning is displayed on the status line if they are not the same. This can be very useful when replacing strings in binary files.

Find pattern (old egrep style)

The pattern to find. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, and entering unprintable and special characters and inserting pattern match characters.

See also

  • An Introduction to Pattern Matching

  • Example patterns
  • Pattern matching reference Note: This pattern type does NOT support perl-style {...} quantifiers, or perl-style (?:....) brackets.

Pattern options button [...]

Maximum match size

Changes the maximum search buffer size. If you need to match a string that is longer than 4096 bytes (4k), increase the Maximum text buffer size to be at least as large as the string you need to match.

Match Case

When checked, the case of the string found must match the case of the Find Pattern exactly.

Find Whole Words Only

Only match the Find Pattern when it is not part of a larger word e.g. find "the" but not "them", "there", "worth" etc.

Replace With

The text to replace the found text with. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering unprintable and special characters and inserting pattern replacement specifiers:

$0 - is replaced by the entire matched expression
$1 - is replaced with the first parenthesised expression.
$2 - is replaced with the second parenthesised expression
$3 - $9, $a - $z and so on.

If you need to put a dollar sign $ in the Replace With section, simply double it i.e. $$.

The replacement text is passed to any sub filters.

Case Sensitive Replace

If the Find Pattern matches text that is entirely in UPPERCASE, this forces the replace string to all UPPERCASE. If the Find Pattern matches text that is entirely in lowercase, this forces the replace string to all lowercase.

Replace First Only

Only replace the first occurrence found - skip all remaining matches.

Prompt On Replace

By checking this option, you can manually verify each match before it gets replaced. You can choose to replace or ignore single matches, the remainder of the file or the entire job.

Skip Prompt If Identical

If the find string and replace string are identical, this option can be enabled to skip prompting. This can happen when the case of the search string is identical to the case of the replace string during a case-insensitive search.

Extract matches

When this option is checked all non-matching text will be discarded, leaving only the replacement text. This can be handy for data mining content from web sites or data files.


Perl Pattern Search and Replace

 Perl Pattern Search and Replace

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Use this filter to find a text pattern. You can also use this filter to find multi-line text.

The length of each string is shown above it, and a warning is displayed on the status line if they are not the same. This can be very useful when replacing strings in binary files.

Find pattern (perl-style)

The pattern to find. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, and entering unprintable and special characters and inserting pattern match characters.

See also

  • An Introduction to Pattern Matching
  • Example patterns
  • Pattern matching reference

Pattern options button [...]

Default Match Length ("Greediness")

When set to maximal match, TextPipe will try to match the pattern against the longest piece of text possible. When set to minimal match, TextPipe will try to match the pattern against the shortest piece of text possible. Minimal matching is the default and is the most often used, because it prevents the .* operator from matching an end tag at the end of the file rather than the very next one, and for this reason it is generally much faster than a maximal match. With a large maximum text buffer size, greedy matching can be very inefficient.

This option inverts the "greediness" of the quantifiers so that they are not greedy by default, but become greedy if followed by "?". It can also be set by a (?U) option setting within the pattern.

Maximum Text Buffer Size

Changes the maximum search buffer size. If you need to match a string that is longer than 4096 bytes (4k), increase the Maximum text buffer size to be at least as large as the string you need to match.

Technical Notes:

The pattern matching engine is given a block of text to work with. This block of text is guaranteed by TextPipe to be at least 4K larger than the size of the "Maximum Text Buffer Size" - the block may be larger depending on what filters precede the pattern match filter. Let's call this buffer size the 'critical size'. This critical size allows for efficient matching, because if the buffer is too large, the pattern matching engine may have to check many cases before it fails, so it is more efficient to make it fail earlier by having a smaller buffer size. If greedy matching is turned on, having a larger buffer size may make matching very inefficient.

Allow Comments In Pattern

This enables Perl-style comments and extra white space to be included the pattern match. White space data characters in the pattern are totally ignored except when escaped or inside a character class, and characters between an unescaped # outside a character class and the next new line character, inclusive, are also ignored. This is very handy for including comments inside complicated patterns. Note, however, that this applies only to data characters. White space characters may never appear within special character sequences in a pattern, for example within the sequence (?( which introduces a conditional sub pattern.

'.' Matches Newline

When enabled (default) a dot metacharacter in the pattern matches all characters, including new lines. Without it, new lines are excluded. A negative class such as [^a] always matches a new line character, independent of the setting of this option.

Enable UTF-8 Support

Enables pattern matching in UTF-8 encoded data.

Match Case

When checked, the case of the string found must match the case of the Find Pattern exactly.

Find Whole Words Only

Only match the Find Pattern when it is not part of a larger word e.g. find "the" but not "them", "there", "worth" etc.

Replace With

The text to replace the found text with. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering unprintable and special characters and inserting pattern replacement specifiers:

$0 - is replaced by the entire matched expression
$1 - is replaced with the first parenthesised expression.
$2 - is replaced with the second parenthesised expression
$3 - $9, $a - $z and so on.

If you need to put a dollar sign $ in the Replace With section, simply double it i.e. $$.

The replacement text is passed to any sub filters.

Case Sensitive Replace

If the Find Pattern matches text that is entirely in UPPERCASE, this forces the replace string to all UPPERCASE. If the Find Pattern matches text that is entirely in lowercase, this forces the replace string to all lowercase.

Replace First Only

Only replace the first occurrence found - skip all remaining matches.

Prompt On Replace

By checking this option, you can manually verify each match before it gets replaced. You can choose to replace or ignore single matches, the remainder of the file or the entire job.

Skip Prompt If Identical

If the find string and replace string are identical, this option can be enabled to skip prompting. This can happen when the case of the search string is identical to the case of the replace string during a case-insensitive search.

Extract matches

When this option is checked all non-matching text will be discarded, leaving only the replacement text. This can be handy for data mining content from web sites or data files.


Sounds-like Search and Replace

 Sounds-like Search and Replace

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Use this filter to find a word that sounds like another word, usually to locate a spelling mistake or a spelling variation. Sounds-like (soundex) matching makes surnames such as Smith, Schmidt and Smythe all appear identical.

The length of each string is shown above it, and a warning is displayed on the status line if they are not the same. This can be very useful when replacing strings in binary files.

Find a word that sounds like

The single word to find. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy. No special characters are allowed.

Match Case

Disabled. All words are compared without case sensitivity.

Find Whole Words Only

Disabled - always on by default.

Replace With

The text to replace the found text with. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, and entering unprintable and special characters.

To insert the found text into the replacement, use the text $0. If you need to put a dollar sign $ in the Replace With section, simply double it i.e. $$.

The replacement text is passed to any sub filters.

Case Sensitive Replace

If the Find String matches text that is entirely in UPPERCASE, this forces the replace string to all UPPERCASE. If the Find String matches text that is entirely in lowercase, this forces the replace string to all lowercase.

Replace First Only

Only replace the first occurrence found - skip all remaining matches.

Prompt On Replace

By checking this option, you can manually verify each match before it gets replaced. You can choose to replace or ignore single matches, the remainder of the file or the entire job.

Skip Prompt If Identical

If the find string and replace string are identical, this option can be enabled to skip prompting. This can happen when the case of the search string is identical to the case of the replace string during a case-insensitive search.

Extract matches

When this option is checked all non-matching text will be discarded, leaving only the replacement text. This can be handy for data mining content from web sites or data files.


Unicode Search and Replace (UTF-16LE)

 Unicode Search and replace (UTF-16LE)

Submit feedback on this topic

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Use this filter to find Unicode text that appears exactly as you have typed it, or with a case variation. You can also use this filter to find multi-line text.

The length of each string is shown above it, and a warning is displayed on the status line if they are not the same. This can be very useful when replacing strings in binary files.

Note 1: The Unicode file being processed must be in UTF-16LE form. If it is not, use the Unicode conversion filter to convert it.

Note 2: If you're using the Trial Run area to test the replacement, ensure that the Trial Run Extra Options are set to Treat Trial Input as Unicode (checked) and Treat Trial Output as Unicode (checked).

Find Unicode String

The text to find. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, and entering unprintable and special characters.

Pattern options button [...]

Maximum match size

Changes the maximum search buffer size. If you need to match a string that is longer than 4096 bytes (4k), increase the Maximum text buffer size to be at least as large as the string you need to match.

Ignore non-spacing characters

Ignores characters that have zero width

Space compression

Treats multiple spaces and tabs the same as a single space

Match Case

When checked, the case of the string found must match the case of the Find String exactly.

Find Whole Words Only

Only match the Find String when it is not part of a larger word eg find "the" but not "them", "there", "worth" etc.

Replace With

The text to replace the found text with. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, and entering unprintable and special characters.

To insert the found text into the replacement, use the text $0. If you need to put a dollar sign $ in the Replace With section, simply double it i.e. $$.

The replacement text is passed to any sub filters.

Case Sensitive Replace

If the Find String matches text that is entirely in UPPERCASE, this forces the replace string to all UPPERCASE. If the Find String matches text that is entirely in lowercase, this forces the replace string to all lowercase.

Replace First Only

Only replace the first occurrence found - skip all remaining matches.

Prompt On Replace

By checking this option, you can manually verify each match before it gets replaced. You can choose to replace or ignore single matches, the remainder of the file or the entire job.

Skip Prompt If Identical

If the find string and replace string are identical, this option can be enabled to skip prompting. This can happen when the case of the search string is identical to the case of the replace string during a case-insensitive search.

Extract matches

When this option is checked all non-matching text will be discarded, leaving only the replacement text. This can be handy for data mining content from web sites or data files.


Unicode Pattern Search and Replace (UTF-16LE)

 Unicode Pattern Search and replace (UTF-16LE)

Submit feedback on this topic

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up
Unicode Pattern Reference

Use this filter to find Unicode text that appears exactly as you have typed it, or with a case variation. You can also use this filter to find multi-line text.

The length of each string is shown above it, and a warning is displayed on the status line if they are not the same. This can be very useful when replacing strings in binary files.

Note 1: The Unicode file being processed must be in UTF-16LE form. If it is not, use the Unicode conversion filter to convert it.

Note 2: If you're using the Trial Run area to test the replacement, ensure that the Trial Run Extra Options are set to Treat Trial Input as Unicode (checked) and Treat Trial Output as Unicode (checked).

Find Unicode Pattern

The pattern to find. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, and entering unprintable and special characters.

See also

  • Unicode patterns reference

Pattern options button [...]

Maximum match size

Changes the maximum search buffer size. If you need to match a string that is longer than 4096 bytes (4k), increase the Maximum text buffer size to be at least as large as the string you need to match.

Ignore non-spacing characters

Ignores characters that have zero width

Space compression

Treats multiple spaces and tabs the same as a single space

Match Case

When checked, the case of the string found must match the case of the Find String exactly.

Find Whole Words Only

Only match the Find String when it is not part of a larger word eg find "the" but not "them", "there", "worth" etc.

Replace With

The text to replace the found text with. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, and entering unprintable and special characters.

To insert the found text into the replacement, use the text $0. If you need to put a dollar sign $ in the Replace With section, simply double it i.e. $$.

The replacement text is passed to any sub filters.

Case Sensitive Replace

If the Find String matches text that is entirely in UPPERCASE, this forces the replace string to all UPPERCASE. If the Find String matches text that is entirely in lowercase, this forces the replace string to all lowercase.

Replace First Only

Only replace the first occurrence found - skip all remaining matches.

Prompt On Replace

By checking this option, you can manually verify each match before it gets replaced. You can choose to replace or ignore single matches, the remainder of the file or the entire job.

Skip Prompt If Identical

If the find string and replace string are identical, this option can be enabled to skip prompting. This can happen when the case of the search string is identical to the case of the replace string during a case-insensitive search.

Extract matches

When this option is checked all non-matching text will be discarded, leaving only the replacement text. This can be handy for data mining content from web sites or data files.


Unicode Pattern Reference

 Unicode Pattern Reference

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Regular expression search engine for text in UCS2 form taking surrogates into account. This implementation is an improved translation from the URE package written by Mark Leisher who used a variation of the RE->DFA algorithm done by Mark Hopkins.

Assumptions

  1. Regular expression and text already normalized
  2. Conversion to lower case assumes a 1-1 mapping.

Definitions

Separator - any one of U+2028, U+2029, NL, CR.

Operators

  • . - match any character
  • * - match zero or more of the last sub expression
  • + - match one or more of the last sub expression
  • ? - match zero or one of the last sub expression
  • () - sub expression grouping
  • {m, n} - match at least m occurrences and up to n occurrences
    Note: both values can be 0 or omitted which denotes then a unlimited bound {,} and {0,} and {0, 0} correspond to * {, 1} and {0, 1} correspond to ? {1,} and {1, 0} correspond to +
  • {m} - match exactly m occurrences

Notes

The "." operator normally does not match separators, but a flag is available that will allow this operator to match a separator.

Literals and Constants

  • c - literal UCS2 character
  • \x.... - hexadecimal number of up to 4 digits
  • \X.... - hexadecimal number of up to 4 digits
  • \u.... - hexadecimal number of up to 4 digits
  • \U.... - hexadecimal number of up to 4 digits

Character classes

  • [...] - Character class
  • [^...] - Negated character class
  • \pN1,N2,...,Nn - Character properties class
  • \PN1,N2,...,Nn - Negated character properties class

POSIX character classes recognized: :alnum: :alpha: :cntrl: :digit: :graph: :lower: :print: :punct: :space: :upper: :xdigit:

Notes

  • Character property classes are \p or \P followed by a comma separated list of integers between 0 and the maximum entry index in TCharacterCategory. These integers directly correspond to the TCharacterCategory enumeration entries. Note: upper, lower and title case classes need to have case sensitive search be enabled to match correctly!
  • Character classes can contain literals, constants and character property classes. Example:

    [abc\U10A\p0,13,4]


MS Word Pattern Search and Replace

 MS Word Pattern Search and Replace

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter is identical in function to a pattern search and replace, except that it supports MS Word-style syntax for specifying patterns:

To find | Use this wildcard | Examples
Any single character | ? | s?t finds "sat" and "set".
Any string of characters | * | s*d finds "sad" and "started".
One of the specified characters | [ ] | w[io]n finds "win" and "won".
Any single character in this range | [-] | [r-t]ight finds "right" and "sight". Ranges must be in ascending order.
Any single character except the characters inside the brackets | [!] | m[!a]st finds "mist" and "most", but not "mast".
Any single character except characters in the range inside the brackets | [!x-z] | t[!a-m]ck finds "tock" and "tuck", but not "tack" or "tick".
Exactly n occurrences of the previous character or expression | {n} | fe{2}d finds "feed" but not "fed".
At least n occurrences of the previous character or expression | {n,} | fe{1,}d finds "fed" and "feed".
From n to m occurrences of the previous character or expression | {n,m} | 10{1,3} finds "10", "100", and "1000".
One or more occurrences of the previous character or expression | @ | lo@t finds "lot" and "loot".
The beginning of a word | < | <(inter) finds "interesting" and "intercept", but not "splintered".
The end of a word | > | (in)> finds "in" and "within", but not "interesting".

Notes

  • Type <(pre)*(ed)> to find "presorted" and "prevented".

  • You can use the $n wildcard to search for an expression and then replace it with the rearranged expression. For example, type (Newton) (Christie) in the Find what box and $\2 $1 in the Replace with box. TextPipe will find "Newton Christie" and replace it with "Christie Newton".

MS Word Pattern Troubleshooting

Type the search text correctly Make sure you didn't mistype the text or add extra spaces before or after it.

Clear the Match case check box If this check box is selected, TextPipe finds only text with the specified capitalization (for example, if you search for "Most", TextPipe won't find "most" or "MOST").

Clear the Find whole words only check box If this check box is selected, TextPipe finds only complete words (for example, if you search for "enter", TextPipe won't find "entertain").

When I use MS Word wildcard characters, TextPipe doesn't find certain text.

Examples For example, a search for s*t will find "sat" but not "Sat" or "SAT". If you want to search for any combination of uppercase and lowercase letters, use the square brackets wildcard. For example, type [Ss]*[Tt] to find "sat", "Sat", or "SAT".

Specify all variations of a word When you use wildcards, TextPipe finds whole words only. For example, a search for e*r will find "enter" but not "entertain". (Notice that the Find whole words only check box is dimmed to indicate that this option is automatically on; you can't turn off the option.) If you want to search for all variations of a word, use the appropriate wildcard characters. For example, type <(e*r) to find "enter" and "entertain".


Restrict to each line in turn

 Restrict each line in turn

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter restricts sub filters to operate on each line in turn. This filter is used for its side effect of limiting the matched text to a single line at most.

See also

Sub filters
Restriction filters


Link to TextPipe filter file (.fll)

 Link to TextPipe filter file (.fll)

Submit feedback on this topic

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter incorporates the filters found in an external file. By using the feature, you can build small filters that perform useful sub-functions, and then use the linking feature to embed them in one larger filter. Changes to the subfilter will be reflected in all filters that link to it.

For example, you might have a filter than you always use for formatting paragraphs. You might link to it in other filter that reformat pages, number lines or generate lists. You use the Link Filter capability to generate reusable chunks of filters.


XSLT Filter

 XSLT Filter

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

This filter applies an XSLT transformation to XML text. The filter depends on MSXML, http://www.microsoft.com/downloads/details.aspx?FamilyId=C0F86022-2D4C-4162-8FB8-66BFC12F32B0&displaylang=en. Below is a sample XSLT script, to convert XML data to SQL INSERTscript format.

<?xml version="1.0"?>
<!--
XSL Stylesheet
Transforms NewMMARS feedback files into SQL
-->
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
<xsl:output method="text"/>

<xsl:template match="//AMS_DOC_XML_EXPORT_FILE">
<xsl:text>update mmars_return_file set type = 'VCC' where mmars_return_file_id
  = IDENT_CURRENT('mmars_return_file');&#13;&#10;</xsl:text>
<xsl:apply-templates select='AMS_DOCUMENT'/>
</xsl:template>

<xsl:template match="/AMS_DOC_XML_EXPORT_FILE/AMS_DOCUMENT">
<xsl:if test="VC_DOC_VCUST/DOC_DEPT_CD='TRE' and VC_DOC_VCUST/DOC_UNIT_CD='1402'">
<xsl:text>insert into mmars_vcc (mmars_return_file_id, doc_id, vend_cust_cd,
  ad_id) values (IDENT_CURRENT('mmars_return_file'),'</xsl:text>
<xsl:value-of select="VC_DOC_VCUST/DOC_ID"/><xsl:text>', '</xsl:text>
<xsl:value-of select="VC_DOC_VCUST/VEND_CUST_CD"/><xsl:text>', '</xsl:text>
<xsl:value-of select="VC_DOC_AD[AD_TYP='PA']/AD_ID"/><xsl:text>');&#13;&#10;</xsl:text>
</xsl:if>
</xsl:template>

<xsl:template match="//AMS_IC_STATUS">
<xsl:text>update mmars_return_file set type = 'VSS' where mmars_return_file_id
  = IDENT_CURRENT('mmars_return_file');&#13;&#10;</xsl:text>
<xsl:apply-templates select='AMS_DOCUMENT'/>
</xsl:template>

<xsl:template match="AMS_DOCUMENT">
<xsl:if test="DOC_DEPT_CD='TRE' and DOC_UNIT_CD='1402'">
<xsl:text>insert into mmars_vss mmars_return_file_id, batch_id, doc_id, doc_cd,
  doc_vers_no, doc_func_cd, doc_phase_cd, doc_sta_cd) values (CURRENT_IDENT('mmars_return_file'),
  '</xsl:text>
<xsl:value-of select="@BATCH_ID"/><xsl:text>', '</xsl:text>
<xsl:value-of select="DOC_ID"/><xsl:text>', '</xsl:text>
<xsl:value-of select="DOC_CD"/><xsl:text>', '</xsl:text>
<xsl:value-of select="DOC_VERS_NO"/><xsl:text>', '</xsl:text>
<xsl:value-of select="DOC_FUNC_CD"/><xsl:text>', '</xsl:text>
<xsl:value-of select="DOC_PHASE_CD"/><xsl:text>', '</xsl:text>
<xsl:value-of select="DOC_STA_CD"/><xsl:text>');&#13;&#10;</xsl:text>
</xsl:if>
</xsl:template>

</xsl:stylesheet>

See also

Database filter


Inserting data into a database

 Inserting data into a database

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Using TextPipe you can insert processed data directly into a database without having to run a second program.

The database filter assumes that incoming text is one or more SQL commands, each separated by a semi-colon. To insert data into a database, the text generally looks like this:

insert into products ( product_id, productname ) values ( 188875, 'AbusePipe Single User License' );
insert into products ( product_id, productname ) values ( 146467, 'Annual Renewal' );
insert into products ( product_id, productname ) values ( 139425, 'Annual Renewal (18%)' );

(This was created using a database filter set to generate a SQL Insert Script for table 'products', with Trial Run Input of 'select product_id,productname from products')

In the example above we are inserting into the 'products' table, into the fields product_id and productname.

Let's assume that our data is in CSV (Comma Separated Value) form:

188875,AbusePipe Single User License
146467,Annual Renewal
139425,Annual Renewal (18%)

We need to make some simple changes to it to manipulate it into the SQL Insert Script form.

  1. String fields usually need to be placed inside single or double quotes (depending on your database). This can be done using the Restrict to delimited fields (CSV, Tab, Pipe etc), with subfilters of an Add Header and Add Footer filter containing a single or double quote.

  2. Use an Add Left Margin filter with text of

    insert into products ( product_id, productname ) values (

  3. Use an Add Right Margin filter with text of

    );

  4. Add a database filter for the database the table is in.

Tutorial Index


Sample filters

 Sample filters

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

TextPipe comes with over 100 sample filter list files (.fll files) that solve common text processing problems.

Opening a sample filter

To open a sample filter:

  1. Click on Open in the File menu
  2. TextPipe displays the filter list files in the TextPipe folder, usually c:\Program Files\TextPipe. Navigate to the TextPipe folder if the folder displayed is not the TextPipe folder
  3. Select a filter list file (.fll file) to open, or navigate into one of the subject area subfolders such as coding, conversions, email, etext, fun, html, sysadmin or vbscript and select a filter list file to open
  4. Click the Ok button
  5. TextPipe will open and display the selected filter list file.

You can now add, delete or modify filters in the displayed filter list.

Inserting a sample filter

Sometimes you need to incorporate the filters that you have saved in a filter list file. To copy them in to the current filter:

  1. Position the cursor on top of the filter where you want the inserted filter list to appear. The inserted filter list will appear ABOVE the currently selected filter
  2. Click Insert in the File menu
  3. TextPipe displays the filter list files in the TextPipe folder, usually c:\Program Files\TextPipe. Navigate to the TextPipe folder if the folder displayed is not the TextPipe folder
  4. Select a filter list file (.fll file) to insert, or navigate into one of the subject area subfolders such as coding, conversions, email, etext, fun, html or sysadmin and select a filter list file to insert
  5. Click the Ok button
  6. TextPipe will insert the new filter list above the currently selected filter.

Linking to an existing filter (Pro version only)

Any changes made to an inserted filter list file will not be reflected in the main filter list after is has been inserted. TextPipe provides a linking facility that specifies the file name of an existing filter list file to insert, and the insertion is only performed at the time the filter list is used. This means than any changes that have been made to the linked filter list will be reflected in any filter list that links to it. To link to a filter list:

  1. Position the cursor on top of the filter where you want the linked filter list to appear. The linked filter list will appear BELOW the currently selected filter
  2. Click Link to filter in the File menu (Pro version only)
  3. TextPipe displays the filter list files in the TextPipe folder, usually c:\Program Files\TextPipe. Navigate to the TextPipe folder if the folder displayed is not the TextPipe folder
  4. Select a filter list file (.fll file) to link to, or navigate into one of the subject area subfolders such as coding, conversions, email, etext, fun, html or sysadmin and select a filter list file to link
  5. Click the Ok button
  6. TextPipe will insert a new link filter BELOW the currently selected filter.


Step by step

 Step by step

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up
Step 2a. Process One or More Files
Step 2b. Process the Windows Clipboard
Step 3. Filters to Apply
Step 4. Where Should the Output Go?
Step 5. Run the Filter

Where is the data that you want to process?

  1. in one or more files
  2. on the windows clipboard

Note: You can also use the Filter Wizard [image] , found in the Tools menu, to step you through this process.


Step 2a. Process One or More Files

 Step 2a. Process One or More Files

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Next Step >

  1. Select the Files to process tab
  2. You need to add the files to be processed to this window. There are several methods of doing so:
    1. Drag and drop files and folders from Windows Explorer
    2. Click the Add Folder button to select an entire folder, or to select files from your Network
    3. Click the ellipsis button [...] to display a window in which you can select one or more files (press [Ctrl] to select extra files)
    4. Click the [Open] icon to open a file list from an existing file
    5. Simply type the required filename in the Filename box. The Filename cell supports the standard wildcards that the Windows Find dialog supports.
  3. In the TextPipe window, select the Filters to Apply tab
  4. Select the File Input or Clipboard input filter near the top of the filter list. The right side of the screen changes to display the options for the input filter
  5. Ensure that the Input from file(s) radio button is selected. Input is now set to files to come from the files you have selected.

Note: New rows are automatically added at the base of the grid as entries are added.

Including Subfolders

You can ask TextPipe to scan subfolders by setting the Subfolders column to All, or to ignore Subfolders by setting it to None.

How the Rows are Processed - Include and Exclude

Each row is part of a group of files that are processed together. You can include and exclude files from a group using the Actions Include and Excluce. The exclude action only affects files in the current group, so you can remove files from one file group without affecting similar files in other groups.

For example, lets say you had to process a number of documents in the folders c:\My Documents and n:\clients\documents. Unfortunately, there are some invoices in the c:\My Documents folder that should not be processed, but there are invoices in the n:\clients\documents folder that should be processed. By using a file group, you can tell TextPipe to only remove the invoices in the first folder but not in the second folder. Your file list would look like this:

Note: This screen shot shows an older version of TextPipe where Add=Include and Subtract=Exclude.

Next Step >


Step 2b. Process the Windows Clipboard

 Step 2b. Process the Windows Clipboard

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Next Step >

  1. In the TextPipe window, select the Filters to apply tab
  2. Select the File Input or Clipboard input filter near the top of the filter list. The right side of the screen changes to display the options for the input filter
  3. Ensure that the Input from clipboard radio button is selected.

Next Step >


Step 3. Filters to Apply

 Step 3. Filters to Apply

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Next Step >

Now you must decide what filters to apply to the text. Each filter performs an operation such as a search and replace, adding text to the left margin, converting end of line characters etc, and then passes the result on to the next filter.

The Filters to Apply tab controls which filters get applied and the order that they are applied in. To add a new filter to the list:

  1. Place the cursor ABOVE the position where you would like the new filter. The new filter will be added below the cursor
  2. Choose a filter to add from the filter menus (Convert, Add, Remove, Email, Replace, Special, Restrict, Maps)
  3. The new filter will be added below the cursor
  4. Edit the filter's properties on the right hand side of the window.

Below are some common filtering operations and where to find the appropriate filter in the menus.

Common operations | Filter to use:
Convert End of Line characters between Unix/DOS/Mac | Convert Menu\End of Line characters
Search and replace | Replace Menu\Find exactly
Pattern match and replace | Replace Menu\Find pattern
Add left margin (indent) | Add Menu\Left margin
Remove HTML | Remove Menu\HTML and XML
Delete columns | Remove Menu\Columns
Extract email addresses for mailing lists | Email Menu\Extract emails (Pro version only)
Remove blank lines | Remove Menu\Blank lines
Sort | Special Menu\Sort

After the filters have been added to the filter list, you can click on them to display options in the right hand side of the window. Each filter can be tailored in this manner.

Next Step >


Step 4. Where Should the Output Go?

 Step 4. Where Should the Output Go?

Submit feedback on this topic

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up
Step 4a. Overwrite the Original Files
Step 4b. Output to a New Folder
Step 4c. Output to a New File Extension
Step 4d. Output to a Test Run
Step 4e. Output to the Windows Clipboard
Output to a new file by joining all files together

Where should the output be sent?

  1. Output to a test run
  2. Output back to the original files (overwrite them)
  3. Output to a new folder
  4. Output to a new file extension
  5. Output to the windows clipboard
  6. Output to a new file by joining all files together


Step 4a. Overwrite the Original Files

 Step 4a. Overwrite the Original Files

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Next Step >

  1. In the TextPipe window, select the Filters to apply tab
  2. Select the File Output or Clipboard ouput filter at the end of the filter list. The right side of the screen changes to display the options for the ouput filter
  3. Ensure that the Output to File(s) radio button is selected
  4. Ensure that Test Mode, Change extension to and Output folder are all unchecked.

Next Step >


Step 4b. Output to a New Folder

 Step 4b. Output to a New Folder

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Next Step >

  1. In the TextPipe window, select the Filters to apply tab
  2. Select the File Output or Clipboard ouput filter at the end of the filter list. The right side of the screen changes to display the options for the ouput filter
  3. Ensure that the Output to File(s) radio button is selected
  4. Ensure that the Output Folder box is checked
  5. Select a new folder by clicking the folder icon on the right hand side of the Output Folder box, or by typing a new folder name
  6. To maintain the existing folder structure, check the Maintain folder structure check box. To remove the folder structure of the original files, leave the box unchecked.

Each input file is written out to a new folder.

Next Step >


Step 4c. Output to a New File Extension

 Step 4c. Output to a New File Extension

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Next Step >

  1. In the TextPipe window, select the Filters to apply tab
  2. Select the File Output or Clipboard ouput filter at the end of the filter list. The right side of the screen changes to display the options for the ouput filter
  3. Ensure that the Output to File(s) radio button is selected
  4. Ensure that the Change extension to box is checked.

The original extension of each file is removed and replaced with the given extension. A leading period is assumed if not provided.

Next Step >


Step 4d. Output to a Test Run

 Step 4d. Output to a Test Run

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Next Step >

  1. In the TextPipe window, select the Filters to apply tab
  2. Select the File Output or Clipboard ouput filter at the end of the filter list. The right side of the screen changes to display the options for the ouput filter
  3. Ensure that the Output to File(s) radio button is selected
  4. Ensure that the Test Mode box is checked.

Each input file is written out to a completely new file, with an extra extension added of .dCv.

Next Step >


Step 4e. Output to the Windows Clipboard

 Step 4e. Output to the Windows Clipboard

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Next Step >

  1. In the TextPipe window, select the Filters to apply tab
  2. Select the File Output or Clipboard ouput filter at the end of the filter list. The right side of the screen changes to display the options for the ouput filter
  3. Ensure that the Output to Clipboard (merge) radio button is selected.

If the input consists of multiple files, they will be added together and sent to the clipboard.

Next Step >


Output to a new file by joining all files together

 Output to a new file by joining all files together

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Next Step >

  1. In the TextPipe window, select the Filters to apply tab
  2. Select the File Output or Clipboard ouput filter at the end of the filter list. The right side of the screen changes to display the options for the ouput filter
  3. Ensure that the Single file Output radio button is selected
  4. Choose the file where the joined output will be sent.

Next Step >


Step 5. Run the Filter

 Step 5. Run the Filter

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Congratulations, your filter is now ready to run!

wpe7.jpg (2525 bytes)

Click the [Go] button to begin processing.

To process text on the Windows clipboard (overriding the Input and Output filter settings) click the [Clipboard] button.

If you wish to test the filter first, select the Trial Run tab, paste or type some text into the Input Text area, then click the [Trial Run] button.


Troubleshooting filters

 Why isn't my filter working?

Submit feedback on this topic

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Try the following suggestions:

  1. Use the partial trial run facility of the Filters to Apply tab to either run individual filters or all filters up to a given point

  2. No files processed? Are you processing binary files? If so, check that the Input Filter has the Binary Files option set to 'Process' or 'Confirm before processing'. The status dialog also shows a count of the Binary files skipped in Red so it is easy to tell.
  3. Where is your output going? Check that the settings for Test Mode, Change extension to, Output folder, Clipboard output and Single file output on the output filter are set as you expect.
  4. Which files and folders are you processing? Check the files tab.
  5. Is any input getting through to the output? You can check this by inserting Debug filters at strategic points in your filter to monitor the text at various stages.
  6. You can also use the Filter Options to enable logging - this will allow you to see which files are processed, what gets done to them and where they are saved.

If you're running TextPipe from the command line, try the Command Line Helper and the Show Command Line dialogs, and the command line troubleshooting guide.

Stack Overflow Errors

If your patterns are poorly written (or alternatively, you find a bug in the pattern matching library we use), you may run into Stack Overflow errors. These occur when a pattern keeps matching text without any limit, and eventually runs out of memory. They can be avoided by limiting how much text a pattern can match - see Performance. To find the offending filter, use these steps:

  1. Load the file into the Trial Run area
  2. Right click in the filter list, and choose 'Trial Run up to and including this filter'. Keep altering the filter you click on until the error no longer occurs
  3. The filter you just excluded is the problem filter
  4. In the problem filter, keep deleting the right-hand end of the pattern until no error occurs
  5. The bit you just deleted was the problem. Rewrite it.

You should avoid patterns of the type

([^">])+

where the brackets are unnecessary. Each set of brackets causes a recursive step. This pattern should be written as:

[^">]+

When the match can occur a large number of times, it can often be rewritten as

[^">]{1,10} or ([^">]){1,10}

with no change to the pattern matching process. This avoids too many recursive steps.

If you cannot eliminate the Out of Memory errors or Stack Overflow errors, please send your filter to us.


 Common tasks

Home User Assistance Tutorials How to Use TextPipe
Menus: File Edit Filters[ Convert Add Remove Unicode Replace Special Map Email Restrict ] Tools Window Help Advanced
Home
Up

Common Tasks

Replacing across a website...

  1. Drag and drop your website base/root folder to anywhere in TextPipe's window. It will get added to the Files to Process tab
  2. Add a Replace menu\Find exactly filter for each replacement required
  3. Click the [Go] button.

You can use a variety of pattern matching filters from the Replace menu. If you have more than one replacement, you can use a Replace menu\Search/Replace list, or import a list.

Grep through website logs

  1. Drag and drop your log files or log folder to anywhere in TextPipe's window. They will get added to the Files to Process tab
  2. If required, move the files in the file grid to change the order they are joined (or use a Special Menu\Sort filter)
  3. Add a Special Menu\Merge filter to join the files together
  4. To remove lines you never wish to see, add one or more Remove Menu\Matching lines filter
  5. To extract lines matching a pattern, add a Special Menu\Retain Matching Lines filter (note - any non-matching lines are discarded, so you can't really use 2 Retain Matching Lines filters in a row. Instead, use an or '|' operator in the pattern match)
  6. Set the Output filter to write all data to a new output file
  7. Click the [Go] button.

Fix CSV-delimited or Tab-delimited files

  1. Drag and drop your files or folders to anywhere in TextPipe's window. They will get added to the Files to Process tab
  2. To remove fields, add a Remove menu\Remove delimited fields (CSV, Tab, Pipe etc) filter
  3. To copy fields, add a Special Menu\Copy delimited fields (CSV, Tab, Pipe etc) filter
  4. To move fields, add a Special Menu\Move delimited fields (CSV, Tab, Pipe etc) filter
  5. To replace inside specific fields, add a Restrict Menu\Restrict to delimited fields (CSV, Tab, Pipe etc) filter, then add a Search/Replace sub filter.
  6. To convert specific fields to UPPERCASE, add a Convert Menu\Uppercase filter, then add a Search/Replace sub filter
  7. ..etc..
  8. Click the [Go] button.

Convert from mainframe EBCDIC to ASCII

  1. Drag and drop your files or folders to anywhere in TextPipe's window. They will get added to the Files to Process tab
  2. Add a Convert Menu\EBCDIC -> ASCII filter
  3. Set the Input filter to Process Binary files
  4. Click the [Go] button.

Join, merge and sort files

  1. Drag and drop your files or folders to anywhere in TextPipe's window. They will get added to the Files to Process tab
  2. Add a Special Menu\Merge filter to join the files together
  3. Add a Special Menu\Sort filter
  4. Set the Output filter to write all data to a new output file
  5. Click the [Go] button.

Split files

  1. Drag and drop your files to anywhere in TextPipe's window. They will get added to the Files to Process tab
  2. Add a Special Menu\Split filter to split the files
  3. Set the Input filter to Process Binary files
  4. Click the [Go] button.

Extract data from a database

  1. Add a Special Menu\Database filter. Use the connection string to specify where the database is
  2. In the Trial Run Area, enter the text 'select * from tablename', where tablename is the name of your table
  3. Set the Input filter to Input from Trial Run Input
  4. Set the Output filter to write all data to a new output file
  5. Click the [Go] button.

You may wish to customize the output with additional search/replace filters.

Text and data mining

See the sample filter html\data mine html tables.fll for an example of converting a web page to a CSV file for import into Excel. This uses the Extract option of the Search and Replace filter.

Extract email addresses

  1. Drag and drop your files to anywhere in TextPipe's window. They will get added to the Files to Process tab
  2. Add an Email menu\Extract email addresses filter
  3. Add a Special menu\Merge filter
  4. Add a Special menu\Sort filter
  5. Add a Remove menu\Remove matching lines filter for any addresses you want excluded
  6. Set the Output filter to write all data to a new output file
  7. Click the [Go] button.

Other tasks

To add a filter, choose one from the menu. It will automatically be added to the filter list. Below are some common filtering operations and where to find the appropriate filter in the menus.

Common operations | Filter to use:
Find an HTML tag | Use Replace Menu\Find pattern (perl style). Ensure that pattern matching is enabled using the drop down list, and that Minimal matching is specified in the advanced [...] dialog, then enter a search string like this: <a[^>]*> This will find all <a> tags. The [^>] means 'find any character except a '>', and the * means 'find 0 or more characters like this one.
Find text within an HTML tag e.g. <a....height="130"....> | Use Replace Menu\Find inside an HTML tag. Ensure that pattern matching is enabled using the drop down list, and that Minimal matching is specified in the advanced [...] dialog, then enter a search string like this: <a[^>]*> This will find all <a...> tags. Now add a sub filter to perform the real search and replace on the text found inside the tags.
Find text between HTML tags e.g. <a....>......</a> | Use Replace Menu\Find pattern. Ensure that Minimal matching is specified in the advanced [...] dialog, then enter a search string like this: <a[^>]*>.*</a> This will find all <a>...</a> tags. Now add a sub filter to perform the real search and replace on the text found between the tags.
Remove HTML | Remove Menu\HTML and XML
Retain the head or tail of a file | This is similar to the Unix command head and tail, use the Remove Menu/Remove start or end of file filter to remove lines or characters from the start or end of the file.
Swap date formats from dd-mm-yyyy to yyyy-mm-dd | Using Replace Menu\Find pattern (perl style), add a search pattern of (\d+)-(\d+)-(\d+) and a replacement of $3-$2-$1
Swap name formats from SURNAME, FIRST to FIRST SURNAME | Using Replace Menu\Find pattern (perl style), add a search pattern of (\w+), (\w+) and a replacement of $2 $1
Convert End of Line characters between Unix/DOS/Mac | Convert Menu\End of Line characters
Search and replace | Replace Menu\Find exactly
Pattern match and replace | Replace Menu\Find pattern (perl style)
Add left margin (indent) | Add Menu\Left margin
Unindent | Use the Remove Menu\Columns filter to remove the first two characters on each line
Delete columns | Remove Menu\Columns
Extract email addresses for mailing lists | Email Menu\Extract emails
Remove blank lines | Remove Menu\Blank lines
Sort | Special Menu\Sort

After the filter is added to the filter list, you can click it to display options in the right hand side of the window. Each filter can be tailored in this manner.